Webscraper

Console Web Scraper:

Project Overview

The Console Web Scraper is a C# application I developed using Visual Studio. It allows users to scrape data from three different websites: YouTube, ICTJob, and Bol.com. The scraped data includes video details from YouTube, IT job listings from ICTJob, and product information from Bol.com. The project also features wishlist management for Bol.com products, and the ability to export scraped data to JSON and CSV formats.

Features

• YouTube Scraper: Users can search and retrieve detailed information about videos from YouTube.
• ICTJob Scraper: Users can search and retrieve information about IT job listings from ICTJob.
• Bol.com Scraper: Users can search and retrieve product information from Bol.com.
• Wishlist Management: While scraping data from Bol.com, users can add products to a wishlist for future reference.
• Export Functionality: All scraped data can be exported in either JSON or CSV formats for further analysis or storage.

Getting Started

Prerequisites

• Visual Studio installed.
• .NET Core SDK installed.

Installation

1. Clone the repository to your local machine.
2. Open the solution file (WebScraperConsole.sln) in Visual Studio.
3. Build the solution and run the application.

Usage

1. Launch the application.
2. Choose one of the scraping options from the menu (YouTube, ICTJob, or Bol.com).
3. Follow the prompts to enter search terms and interact with the retrieved data.

Acknowledgments

This project was made possible by utilizing several excellent libraries, including Selenium WebDriver for web scraping, CsvHelper for CSV management, and Newtonsoft.Json for handling JSON data.

Explanation video

Github link

https://github.com/GiosSiebe/Console-Web-Scraper

Conclusion

The Console Web Scraper was a great learning experience in web scraping, C# development, and working with data formats like JSON and CSV. It provided valuable insight into automating data retrieval and manipulation while building practical features like wishlist management.