Community Bonding Period - Week 4
This week I restructured the preliminary code I had previously. I did this to make it work as I explained in my project diagram below. The changes I implemented made it possible to easily download the database.
Later, I worked on making the code reliable because it wasn’t always working in the “headless” mode. There was an undocumented dependency problem in the tor-browser-selenium library that I was using. I needed to install the Firefox browser to reliably use the library. I don’t think it is related to having “Firefox” installed on the system but I think it is related to having a piece of code Firefox installs. It took a long time to figure this out and I will raise this issue in the library’s GitHub repository to further investigate with the maintainers.
I also added the functionality to add new headers to the requests and save response HTTP headers. The original selenium library doesn’t have this functionality, and I needed to find another way to interact with the headers. I ended up using the selenium-wire, which is an extension of the original selenium library and it allowed me to interact with the HTTP headers.
So, I spent this week trying to get a very basic working version of the project, and I did it! Now, I will extend it and make it complete during the coding phase. I think it has been a great community bonding period, which helped me to get feedback from the community and get prepared for the coding phase.