The Selective Car Retrieval And Processing Efficient Robot (S.C.R.A.P.E.R.) was a solo project done for FiWize. Its purpose was to quickly and easily collect data on current makes and models from a list of automotive manufacturers’ websites using web scraping.

The application was written in Python using the open-source Scrapy library. It was written to be compatible with both Windows and Unix operating systems, and was packaged as a single executable to be run from anywhere. The only requirement to run the application is Python itself. Note: the scraping logic was written for each individual website as they were structured in 2021. Websites that have been drastically changed since then will not have their data scraped properly; however the program’s filtering system will exclude incomplete or broken data from being recorded entirely, in order to maintain clean, human-readable data.

To run it for yourself, click the following link, then follow the instructions below.

1. Ensure that you have the Python runtime installed.
2. Extract the .zip file and navigate to <extracted folder>/vehicle_data_tracker/dist
3. Run scraper.exe
4. After S.C.R.A.P.E.R. has finished initializing, select the desired make from the dropdown menu and click “Crawl!”
5. When the process is complete, the results of the crawl will be in the dist directory as vehicles.csv