Experimenting with LLM Insights from Structured Deliberation Graphs

A Small Experiment on Next Steps for The Society Library 🌌 🔭

From the oceanic desk of the Executive Director 🌊

Ahoy from the Atlantic, everyone! 🛟 At the time of writing, you’ll find me (Executive Director of the Society Library, Jamie Joyce) in the middle of the Atlantic ocean as an “Intelligence Fellow” for the Earth One expedition. The idea of this particular trip, as I understand it, is to convene activists and philanthropists with the hope that great collaborations can happen while we’re all at sea. We all won’t see land for 6 days (although the full trip was two weeks long), so I am hopeful we will truly be able to dive deep into each other’s work and find fruitful collaborations.

This is where I am

As a part of the presentation of the “Intelligence Track” last night, a few questions were posed to our cohort, including:

• How do we develop epistemic humility, and value uncertainty?
• How can rational and non-rational intelligences collaborate rather than compete?
• How do we cultivate a mindset that integrates rational and non-rational intelligences?
• How do we expand perception by learning to sense into other ways of knowing?
• In a world increasingly shaped by algorithms, how do we preserve the role of intuition and emotions in our decision making ?
• What does “wise” technology look like and how does it differ from “smart” tech?
• What does it mean to create tech that does not “solve” problems, but create space for more profound epistemic questions?

So naturally, I passed these questions on to the Society Library team so we can query our automated deliberation-mapping system with these questions.

If you have been following our work, you will have seen that we have made progress “grounding” our maps. Meaning that we use our deliberation-mapping technologies to “predict” the debate we expect to find presented in multimodal artifacts, and then we use that data to query and match claims from an index of topic-related content (e.g. books, scholarly works, government documents, podcasts, social media posts, etc.), so we can validate which of these AI-generated claims, arguments, and pieces of evidence actually exist in human-made publications. We have a lot of work left to go, but we are excited about the progress. However, that process tends to be longer and much more expensive, so for the questions above, we are using the quick and cheap version of the system: ungrounded AI predictions only. So we pressed “the button” and made the graphs.

As expected, these graphs were quite large: there were thousands of positions, arguments, examples and counter-arguments that the system was predicting. While this usually suits our purposes quite well, because we want our inquiry process to be exhaustive, it would be too much to ask our fellow Intelligence Track colleagues to read through the graphs as-is. But, I had an early morning idea about using LLMs to derive some insights from the structured data. Here’s how it went, but first — a word about our orientation towards large graphs.

At the Society Library, we know the map is not the t̵e̵r̵r̵i̵t̵o̵r̵y̵ utility.

The Society Library knows that the graphs (or “deliberation maps”) are not the end product. We actually explore many ways to visualize the data and make it accessible. One way is through our interactive “Papers” Interface. Which actually embeds structured graph data into “unpackable” sentences and words.

See video of Papers walkthrough here.

We also create decision-making models, to allow people to work through graphs “step by step.”

See video of Decision-Making Model demo here (still being digitized)

We also have a much more simple interface for generated perspectives and shallow arguments, called Perspectives. Which you can see here in a video (currently being redeployed), but you can also use here (for neuro-tech topics only).

You can also simply search the graphs.

But even these are just interim tools, because we are looking to the long-term when it comes to deriving insights and making decisions on structured data. Our goal is to enable more informed, less biased, and more informed decision-making on civilization-scale issues, and as soon as you start modeling things at the civilization-scale, it becomes obvious that “computing” that amount of complex data may be well beyond human capability and comprehension. At least, without technological assistance (barring of course other epistemological arguments and frames about how one comes to intuit anything).

However, we feel as though we are still at “step one,” and that’s a really important step to do well. We believe that in order to make optimally informed decisions, we first need comprehensive data. We see ourselves as a library first and foremost, and hope that the large civilization-scale deliberation datasets we create can be useful to many decision-making models — especially computational ones. So, building datasets is the work we are engaging in currently…but that doesn’t stop us from experimenting.

So, this is what we did.

We started with the following question my cohort was posed: “How can rational and non-rational intelligences collaborate rather than compete?”

Then I took the structured JSON file of the deliberation graph (which normally gets imported to Debate Map) and instead uploaded it to Claude. This was a smaller map, with only 1,636 nodes.

Admittedly, it’s a busy day and this is a side-quest, so I did not do too much experimentation with prompts. However, this was the first prompt I gave:

From all of this structured data in this JSON, what is the greatest insights you can derive from this set that would be surprising or unique answers to the question “How can rational and non-rational intelligences collaborate rather than compete?”

Here was Claude (3.5 Sonnet)’s answer:

1. The False Dichotomy of Competition vs Collaboration

Perhaps the most provocative insight is that framing the relationship between rational and non-rational intelligences as either “competitive” or “collaborative” may be fundamentally misguided. The evidence suggests these modes of interaction exist on a spectrum and can actually reinforce each other — competition can drive collaboration, and collaboration can sharpen competitive advantages. For example, in environmental monitoring, competitive pressure between AI systems and natural ecosystems has led to enhanced collaborative outcomes, as when AI-driven conservation efforts create evolutionary pressure that increases species’ resilience.

2. The “Cognitive Market” Theory

A novel framework emerges suggesting that rational and non-rational intelligences naturally create something akin to a “cognitive marketplace” where different forms of intelligence can be traded and exchanged. This is evidenced in indigenous knowledge systems where traditional ecological knowledge trades insights with scientific data collection, each enhancing the other’s value rather than replacing it. This suggests collaboration may be better understood as a form of cognitive commerce rather than either competition or cooperation.

3. The Paradox of Incompatibility Driving Connection

Counterintuitively, the very features that make rational and non-rational intelligences seem incompatible may be what enables their most productive collaborations. Their fundamental differences create cognitive “friction points” that generate novel insights neither could achieve alone. This is demonstrated in biophilic design where the tension between engineered systems and natural processes leads to innovations like the Eastgate Centre in Zimbabwe.

4. The “Extended Intelligence” Hypothesis

The data suggests we may need to reconceptualize intelligence itself not as a property of individual entities but as an emergent phenomenon of interconnected systems. This reframes the question of collaboration from “how can different intelligences work together?” to “how does intelligence manifest across networks of rational and non-rational nodes?” This is evidenced in mycorrhizal networks where tree communities and fungi create distributed intelligence networks.

5. Emotional Intelligence as a Translation Layer

Rather than viewing emotional intelligence as a third type of intelligence to be integrated, the evidence suggests it may function more as an interface or translation layer between rational and non-rational intelligence systems. This is demonstrated in environmental justice initiatives where emotional intelligence helps bridge the gap between quantitative pollution data and qualitative community impacts.

6. The Time-Scale Bridge Theory

A unique insight emerges around how rational and non-rational intelligences may collaborate across different time scales. Rational systems excel at rapid processing and immediate optimization, while non-rational systems often operate on evolutionary or geological time scales. Their collaboration may create bridges across these temporal gaps, as seen in AI-enhanced ecological monitoring that connects real-time data with long-term natural processes.

7. The Ethical Emergence Principle

The data suggests that the ethical frameworks governing rational-non-rational collaboration may need to emerge from the collaboration itself rather than being imposed externally. This is demonstrated in the development of indigenous-led conservation projects where ethical guidelines evolve through the practical work of combining traditional and scientific knowledge systems.

These insights suggest that the key to successful collaboration between rational and non-rational intelligences may not lie in finding ways to make them more similar or compatible, but rather in leveraging their fundamental differences to create emergent capabilities neither possesses alone. This represents a significant departure from conventional approaches to human-AI collaboration or environmental conservation.

Of course, this could be nonsense or these could be interesting insights. So we asked, what informed these insights? This is what it said:

1. The False Dichotomy of Competition vs Collaboration

This insight appears to be primarily derived from two contrasting positions in the JSON:

"position": "Competition, not collaboration, between rational and non-rational intelligences is inevitable and potentially beneficial for overall progress."

Combined with evidence from:

"claim": "Competition between rational and non-rational intelligences could drive more rapid and efficient solutions to environmental challenges..."

And the counterbalancing position:

"position": "Creating adaptive interfaces that bridge the gap between rational and non-rational thought processes can facilitate collaboration."

However, the synthesis suggesting these aren’t actually opposed but exist on a spectrum appears to be an interpretive leap not directly supported by specific examples in the JSON.

2. The “Cognitive Market” Theory

This insight seems to be an extrapolation from examples like:

"example": "The Global Forest Watch platform integrates satellite imagery, machine learning, and crowdsourced data to create an interactive map of global deforestation. Its user-friendly interface allows both scientists and local communities to contribute and access information, bridging the gap between technical forest monitoring and on-the-ground conservation efforts."

However, the specific framing as a “cognitive marketplace” is not directly present in the source material.

3. The Time-Scale Bridge Theory

This insight is more directly supported by the JSON data:

"claim": "[for reasons like] Climate modeling initiatives frequently integrate machine learning algorithms with human expertise, illustrating how shared climate mitigation goals can foster cooperation between computational and human intelligences."

"example": "The Climate CoLab platform, developed by MIT, combines climate models with crowdsourced ideas from citizens, scientists, and policymakers. This interface adapts to both scientific data and public perceptions, fostering collaborative development of climate strategies across diverse stakeholders."

4. Emotional Intelligence as Translation Layer

This concept appears to be derived from:Cop

"claim": "[for reasons like] Emotional intelligence in environmental decision-making can enhance our ability to recognize and respond to subtle ecological changes, complementing rational analysis of quantitative data in climate and biodiversity monitoring."

However, its specific characterization as a “translation layer” represents an interpretive synthesis not directly stated in the source material.

5. The Ethical Emergence Principle

This insight appears to be based on:

"claim": "[for reasons like] Integrating diverse ethical perspectives, including both rational and intuitive approaches, can foster a more inclusive decision-making process that enhances collaboration across different cultural and philosophical backgrounds."

Analysis of Missing Evidence

Several of the insights presented in the summary, while interesting, appear to be interpretive syntheses rather than directly supported claims from the source material:

1. The “Cognitive Market” Theory is a novel framework not explicitly present in the JSON
2. The “Paradox of Incompatibility Driving Connection” represents an interpretive leap
3. The “Extended Intelligence Hypothesis” as described goes beyond the specific examples provided in the source material

Conclusion

While the insights presented are thought-provoking, many represent synthetic interpretations that go beyond the direct evidence provided in the JSON data. The most strongly supported insights are those relating to:

• The integration of emotional intelligence with rational decision-making
• The bridging of different time scales in environmental monitoring
• The emergence of ethical frameworks through practical collaboration

The other insights, while potentially valuable, should be considered more speculative interpretations rather than direct findings from the source material.

Tsk tsk Claude! As per usual, LLMs have a long way to go (or they need to be augmented with other tooling, like we do at The Society Library), but it occurred to us that we simply had never tried this experiment before (and of course, we only tried once). However, there are likely hundreds of companies using LLMs to derive insights from structured qualitative data, and so we’re looking forward to finding more collaborations and doing more experiments to see if this will be yet another way for these large graphs (especially once completely ground in evidence and embeddings from artifacts) could improve our civilization’s epistemics, discourse, and decision-making.

If that sounds like work you are doing, we would love to talk!