I Got Tired of Checking 15 AI Sites Every Morning, So I Built a Pipeline That Does It While I Sleep

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

Global Intelligence Graph is a data ingestion engine that keeps an eye on the AI world so I don't have to.

Staying current in AI means opening arXiv, Papers with Code, GitHub, a handful of news sites and several job boards, then copy-pasting the interesting bits somewhere. I dreaded that chore, so I automated it. One command now:

Anyone who tracks the AI ecosystem (students, researchers, recruiters, founders) gets to be a little lazier: the morning tab-hopping becomes opening a folder of CSVs.

This repository contains the data ingestion pipeline for the AI Engineer demo task. It is designed for massive bulk extraction, 24-hour freshness tracking, and resilient LLM extraction using a multi-tier fallback architecture.

pip install -r requirements.txt playwright install Enter fullscreen mode Exit fullscreen mode Configure Environment Variables: Open the.env file in the root directory and add your API keys: GEMINI_API_KEY = your_gemini_key GROQ_API_KEY = your_groq_key DEEPSEEK_API_KEY = your_deepseek_key Enter fullscreen mode Exit fullscreen mode (Note: You can run the pipeline with just one of these keys, but the fallback chain works best when all three are present.) Running the Pipeline To execute the pipeline: