Mario Carpo

Publication History:

“Generative AI, Imitation Learning, and the Automation of Tacit Knowledge.” A+U 646 (July 2024), 104-109 

The text posted here is a preprint draft and is significantly different from the published version. Please only cite from copy in print

My last book to date, Beyond Digital: Design and Automation at the End of Modernity, was published in the spring of 2023.  It was the third in a series of monographs on contemporary design and technology, similar in format and spirit, which I started writing some twenty years ago.  The Alphabet and the Algorithm, published in 2011, interpreted the rise of digital formalism in the 1990s as the extension—and finally the explosion—of the authorial, allographic way of building, by design and by notation, that was invented by Renaissance Humanists in Europe at the end of the Middle Ages.  The Second Digital Turn (2017), mostly inspired by experiments in computational design then ongoing at the Bartlett School of Architecture, where I started teaching in 2014, focused on Big Data and brute force computing as an alternative to the strategies of data compression and mathematical formalization that had been at the core of modern scientific method since its Galilean inception.  In that book I described the style of “excessive resolution,” or hypergranularity, then popular among computational designers, as the sign of a new, post-human technical logic at play; and while I did not mention artificial intelligence by name in that context, that’s because that term referred, back then, to the kind of computer science developed between the invention of the term itself, in 1956, and its demotion in the late 1970s and 1980s.

I started writing Beyond Digital in the months preceding the Covid-19 pandemic, and I wrote most of it during the first pandemic winter of 2020-21, profiting from the time and concentration that the elimination of all travel, and most human interaction, then induced and seconded.  As a result, that book included perhaps more historiographic research than other recent works of mine, but its main subject had been conceived in pre-pandemic times and was already fully developed when the pandemic struck: the next frontier of computational design, I then argued—and I was in good company—would have to tackle construction and fabrication, and deal somehow with the delivery of actual buildings.  The reasoning behind that assumption was, and remains, somewhat self-evident: computational design tools are already way more advanced that the fabrication tools at our disposal; as a result, the “fabrication gap”—the distance between what we notate and calculate, and what we can build, keeps growing.  Therefore, one would argue, it is high time to shift at least some of our technical prowess from the images we create on the screen to the stuff we can make on site—from bits to atoms.  This shift is made even more urgent by the impending climate crisis, by environmental requirements, and by the divergence between the costs of what we build and current social demand, which—across all nations, people, and places—is today, mostly, for affordable housing.

As the pandemic unfolded, however, it became clear that the brutal adjustments it imposed on the world economy were also fostering and nurturing new ways of living and working.  When the global transportation of people, products, natural resources, and industrial goods, which we had long taken for granted, shrunk and almost came to a halt, particularly during the first waves of pandemic lockdowns, civic and economic life found ways to reset, somehow, and get by on a radically different socio-technical basis: an ancestral one based on local production and geographical proximity, but enhanced by a global communication network of unprecedented efficacy—one which, we then found out almost by surprise, made interaction in the flesh often unnecessary.  This in many ways vindicated many tenets of the digital revolution, probed and discussed at length since at least the mid 1990s: after all, we had always posited, almost as an article of faith, that digital mass-customization is cheaper, faster, smarter, and more environmentally sustainable that the mechanical mass-production of standardized industrial products; and that the electronic transmission of information is cheaper, faster, smarter, and more environmentally sustainable than the mechanical transportation of people and goods.  

At the start of the pandemic, out of necessity, distributed robotic manufacturing, and videoconferencing, stepped in when airports, factories, offices, banks, and schools shut down.  And the Internet mostly did the job, and kept the world economy humming—when almost everything else stopped to function.  Nobody liked that, evidently, but the experience was nonetheless conclusive; as the late Bruno Latour was among the first to remark, the pandemic proved that an alternative to the mechanical-industrial, “anthropocenic” mode of production was, by and large, already technically possible, viable, and ready to go mainstream.[1]  And, some started to observe at that time, that was welcome news, in spite of the unwelcome circumstances, as we all know that the “anthropocenic” mode of production, based on the unlimited exploitation of natural and human resources, was doomed—for a number of reason, not limited to climate change.

What a difference a couple of years make.  Airline traffic, we are told, is back to pre-pandemic levels (even though, based on my personal experience, I beg, anecdotally, to disagree),[2] and developers and municipal entities around the world boast the number of new office buildings, bigger and taller than ever before, now breaking ground in all the usual locations—never mind that most office buildings in midtown Manhattan or in the City of London have been empty for most of the last four years.[3]  Our schools and universities are proudly back to teaching in the flesh, and the fancy equipment for remote and hybrid teaching purchased at huge cost during the pandemic lies unused and forgotten at the back of our lecture theatres, when it was not simply decommissioned or discarded.  As a result, university lecturers have resumed their old itinerant habits, and once again they criss-cross the world to deliver talks that during the pandemic they would have posted on YouTube; the carbon footprint of air travel (which airlines had only recently started to publicize, suggesting that travelers pay for its offset) has been quietly removed from our collective awareness: nobody mentions “flight shaming” any more.  Only a few years ago, cyberspace was seen as something that would take shape at some point in the future; briefly renamed the metaverse, cyberspace is now an ephemeral accident of our recent past: that’s where we lived from 2020 to 2022.

And then, something else happened, that nobody—or, in fact, almost nobody—had seen coming.  A new kind of artificial intelligence, suitably called Generative AI to distinguish it from the failed experiments of the 1960s, irrupted into design schools in the spring and summer of 2022—and into the public realm only a few months later, with the release of the first AI chatbots.  GANs, a transitional mode of generative machine learning, had been known to digital artists since the late 2010s, and design schools started to experiments with GAN created images around 2020: I saw the first GAN projects at the Bartlett School of Architecture, and in Jeff Huang’s studio at the Polytechnical School in Lausanne, during the first crits in remote learning, in the summer or fall of 2020, but I am certain that similar experiments were at time being conducted elsewhere, too; to my knowledge, Matias del Campo was the first critic to relate GAN-based “neural style transfer” to the history of the idea of style in modern architectural theory, from the 19th century to the present.[4] But the real tsunami of Generative AI came with large language models and cross-modal (text-to-image) tools in the summer of 2022; with that, also came the uncritical, generalized adoption of user-friendly image-making interfaces that, at the time of writing (spring 2024), are already a ubiquitous (if often disparaged) feature of design studios in many schools of architecture, and in professional offices around the world.

The practical applications of AI-driven image-processing tools are, for the time being, mostly limited to sketching and renderings—albeit new and unexpected developments may occur anytime, as we have seen time and again in the course of the last two years.  The conceptual, theoretical and, so to speak, methodological implications, and side effects, of the rise of AI-generated images are, however, already momentous: as I have argued elsewhere, Generative AI, either based on legacy GANs or on more recent large language models, is, in epistemological terms, a machine that automates imitation; it is a powerful—albeit purely technical—reminder of a basic, universal condition of human creativity: namely, that nothing is created out of nothing, but all is generated out of something that already exists.  Just like Generative AI can only invent new things based on—but in fact derived from—a dataset of precedents, so all human invention inevitably entails a component of creative imitation: whereby “imitation” means, as in the Greek and Roman, humanistic and classical tradition, the awareness, acknowledgement, election, selection, assimilation, and intelligent transformation, of a corpus of models.  This is a mode of invention that was intensively theorized, and largely practiced, in the European, classical tradition (and, I have reason to assume, in other pre-industrial, artisan traditions around the world) before the rise of industrial modernism, and which modernism eliminated for practical and ideological reasons.  As a result, today nobody knows what imitation is, what it does, and how it works.  And, in so far as Generative AI is an imitation machine, we urgently need to learn again at least the basic of the arts and sciences of imitation, so we know what Generative AI does—and what we do, when we use it.   In short, we need to engage with imitation again, in critical and creative terms; after a century of abeyance and modernist mandated oblivion, we need to come up with a new theory of mimesis.[5]

As a classicist by training—albeit not by any choice of mine, and in very different times—I plaud this development.  So do many colleagues of mine, older and younger, who have similarly been pleading for the inevitability of the reference to precedent in architectural design, and in the visual arts in general.  But, when all is said and done, this is still all and only about images.  Sure enough, images are the main tool of our trade:  no architect would ever claim otherwise.  But, right now, don’t we have bigger fish to fry?  Climate change and social justice require—alongside political choices—technical innovation, and design solutions.  Materials, energy, and the socio-economic consequences of the way we build—which materials we choose, where we take them from, how they are assembled, by whom, and at what cost—all this would appear to matter more, right now, than style and images.  Seen in this light, and in the bigger scheme of things, the recent resurgence of technically driven image-making in architectural design would appear to be, once more, a diversion from more urgent tasks that behove us.

It does not have to be that way.  The core technical logic of Generative AI—the way new inventions are derived from the imitation of a dataset of chosen precedents—is generic, and does not depend on what datasets are made of.  If datasets are made of texts, images, or pairs of texts and images, Generative AI will generate new images or texts based on verbal or visual prompts.  This is how the almost miraculous deeds of Generative AI first struck a chord with the general public.  But dataset, as the name suggests, are made of data.  If our data are, for example, notations of temporal sequences of three-dimensional operations or motions in space, Generative AI will produce new sets of motions derived from those the system was trained on.  Roboticists could then teach robots to imitate sequences of manual operations by sheer observation, or machine learning, in the absence of any specific, rule-based programming; this, evidently, would be invaluable when dealing with operations that cannot be formally scripted, for whatever reason. 

Made for the factory floor, industrial robots were originally designed to re-enact (or “play back”) recorded sequences of exactly repeatable motions.  As the factory floor is a controlled environment, and industrial components are meant to be always the same, industrial robots do not need, in theory, any degree of intelligence, or adaptivity to unforeseen circumstances.  But industrial robots of that ilk cannot pick up tomatoes in a field, because tomatoes on their vine—unlike metal sheets out of a industrial grade press—are all different from one another; industrial robots cannot make pizza, either, because each dough ball will turn out differently when pressed and even more when baked.  And industrial robots are famously useless in building, because building—in spite of one century of mostly failed attempts at industrialization—must to this day deal with many non-standard components, to be put together in endlessly unpredictable ways (with some noted exceptions: bricks, for example).  This is where imitation learning (also known as transfer learning, or learning from example) could be a game changer.  Imitation learning could, in principle, automate artisanal practices entailing operations that do not depend on formalized rules, or formalizable processes. 

And indeed, some experiments and breakthroughs in this direction are already known: Google’s DeepMind team, which famously garnered its expertise by teaching computers to learn games, recently trained a robot to learn how to sort a set of colored blocks by observation alone—i.e., by feeding the system a dataset showing examples of said sorting, which the computer learned to replicate (i.e., imitate) in the absence of any formal rule telling the system what to do, or how to do it. [6]  Likewise, a team of researchers at the University of Pennsylvania and Tongji University has used Generative AI to imitate a dataset of actual and simulated variations of the structure of dragonfly wings—a natural example of very efficient use of materials, but one we cannot optimize using traditional analytic tools.  Imitative variations of that dataset were then further adapted to different geometries and boundaries, including those of airplane wings.[7]

In the building trades, knowledge transfer by imitation learning could serve to replicate the kind of informal artisan skills that some have called “tacit knowledge,” meaning embodied know-how that can not easily be spelled out, nor transmitted, via formulas or rules.  But, as the dragonfly example shows, automated learning from observation could also replace the analytic methods of modern engineering by providing, in some cases, better means to the same end—even leading to solutions that modern, rule-based, deductive tools could not otherwise come up with.   When we cannot calculate, we can still imitate. 

Is this not what pre-industrial artisans always did, before the rise of the mathematically driven, quantitative methods of industrial modernity?   Artisans of old used intuitive, analogic tools out of necessity.   Pre-industrial builders knew how to deal with the materials and the labor force they had locally available, because they had no way to make materials, and workers, come from elsewhere.   But as traditional materials and work processes were by definition almost immutable over time, traditional artisans had no incentive to invent anything new either: the imitation of precedent was all old artisan makers had, because that was all they needed.   Then industrial modernity came, and with it alien, unprecedented materials and techniques; new products and processes that nobody knew what to do with, because they came from unfamiliar factories and distant places.   With them came formulas, handbooks, manuals and instructions; university trained engineers started to draft construction drawings and on site, blueprints sent from far away replaced locally transmitted lore.  As we know, that system worked very well for quite some time.  But as we also know, that system has now stopped working, and we must urgently find some other way to make the things we need, before the implosion of the old way of making ends in disorder and in disaster.   Therefore, it does offer some glints of hope that, just as the modern way of making is conspicuously failing us, Generative AI appears to be offering us a viable, automated, and scalable way of bringing some old ways of making back to life—and with them, hopefully, some of their ancestral, physiocratic sustainability.   

[1]  First in an article Latour published March 30, 2020, on the French digital platform AOC, “Imaginer les gestes-barrières contre le retour à la production d’avant-crise”: https://aoc.media/opinion/2020/03/29/imaginer-les-gestes-barrieres-contre-le-retour-a-la-production-davant-crise. For an English translation, see Latour’s website: http://www.bruno-latour.fr/fr/node/849.

[2] The total volume of airline traffic, measured as the number of paying passengers multiplied by distance traveled, was in 2023 at 94.1 % of the last full pre-pandemic year (2019), and in the month of December 2023 in particular was at 97.5% of the same metric for December 2019 (IATA, press release, 31 January 2024, on line at https://www.iata.org/en/pressroom/2024-releases/2024-01-31-02/).   But, even it’s unlikely that any published data will confirm that, most travelers will have noticed significant shifts in post-pandemic travel patterns: for example, business class seems to be the travel mode of choice, these days, for families with kids, or elderly couples going on holiday abroad.  Business people appear to have learned during the pandemic that Zoom is cheaper.

[3] Carpo, “The Office Was Once a Vital Technology, but Its Time May Be Over,” The Architect’s Newspaper, New York (electronic publication, March 2022:  https://www.archpaper.com/2022/03/op-ed-office-was-once-a-vital-technology-but-its-time-may-be-over/).

 [4]  First in Matias del Campo, Neural Architecture. Design and Artificial Intelligence (Novato, CA: Oro Editions, 2022)

[5]  Carpo, “Formal Analysis, Generative AI and the Eternal Return of Precedent,” Log 58 (October 2023), 133-139;  “Imitation Games. Mario Carpo on the New Humanism,” Artforum 61, 10 (summer 2023), 184-188.

 [6] Scott Reed, et al., “A Generalist Agent,” Transactions on Machine Learning Research, 11/2022, arXiv:2205.06175

 [7] Hao Zheng, Masoud Akbarzadeh, “The Dragonfly Wing Project,” AD 92, 3 (2022), Machine Hallucinations: 132-133.