Dogsheep

Dogsheep is a collection of tools for personal analytics using SQLite and Datasette.

Big internet companies know a lot about us. By exporting that data back out of them we can see what they know and maybe learn something interesting about ourselves.

Read more about Dogsheep on my blog: simonwillison.net/tags/dogsheep

Watch Personal Data Warehouses: Reclaiming Your Data for a demo of Dogsheep in action.

Dogsheep tools

These tools, maintained by the Dogsheep project, let you export your data into a SQLite database for further analysis.

• twitter-to-sqlite uses the Twitter API or Twitter’s “export your data” zip archive to create a database of your tweets, followers, people you follow, favourited tweets and much more besides.
• healthkit-to-sqlite converts your Apple HealthKit export into a database, with details of your heart rate, workouts, step count and more.
• swarm-to-sqlite imports your checkin history from Foursquare Swarm.
• apple-notes-to-sqlite saves Apple Notes data to a database.
• inaturalist-to-sqlite imports the wildlife and plants you have spotted on iNaturalist.
• google-takeout-to-sqlite imports data from Google Takeout, which can include your Google Maps location history, your search history on Google and more to come.
• genome-to-sqlite imports your genome from 23andMe into a SQLite database.
• github-to-sqlite imports repositories you have created or starred on GitHub.
• pocket-to-sqlite imports articles you have saved using Pocket.
• hacker-news-to-sqlite imports stories and comments posted to Hacker News.
• dogsheep-photos imports metadata about photos from Apple Photos, including machine learning labels. See Using SQL to find my best photo of a pelican according to Apple Photos.
• dogsheep-beta is a search index which combines data from all of the other sources into a single faceted search interface.
• evernote-to-sqlite converts exported Evernote files in the ENEX format into a SQLite database.

Tools by other developers

These tools help bring the Dogsheep philosophy to life.

• beeminder-to-sqlite by Ben Congdon saves your goals and data points from Beeminder.
• goodreads-to-sqlite by Tobias Kunze imports your reading history from Goodreads.
• pinboard-to-sqlite by Jacob Kaplan-Moss saves your bookmarks from Pinboard.
• todoist-to-sqlite by Ben Congdon saves your tasks and projects from Todoist.
• parkrun-to-sqlite by Mark Woodbridge imports your parkruns.