Tor Project: Developing Unit, Integration and E2E tests for OONI Probe Desktop apps

Tor Project: Developing Unit, Integration and E2E tests for OONI Probe Desktop apps

About Me

Hello! I am Shivam Shekhar Pathak. I am a pre-final year student of Mathematics and Computing from India.

About My Project

My project involves writing a comprehensive suite of tests for the OONI Probe Desktop applications and automating these tests when a PR is opened to make sure the proposed contributions to the codebase are non-breaking.

My Time Zone

Indian Standard Time (UTC +5:30)

Getting in Touch

GitHub: @eddges
Twitter: @shehges
Mail: sshekhar1159@gmail.com

1 Like

Week 1 and Week 2 (Community Bonding Period)

  • Joined the gsoc-2021 slack channel at OONI.
  • Was warmly welcomed into the OONI community along with another student. Learned about the bi-weekly sprint structure at OONI.
  • Discussed the project plans and the timeline with mentors.
  • Familiarised myself with the libraries and tools required to accomplish the goals of my project.
  • Read about various APIs exposed by Jest and react-testing-library so as to understand possible assertion types.
  • Set up a feedback cycle with my primary mentor @sarath_ms

Some demo tests I wrote during these initial weeks can be found in the repository:

1 Like

Week 3 (May 31 to June 6)

After reading about and understanding the tools and libraries required to work on my project, my goal for this week was to familiarize myself with the current code-base and make configuration changes to facilitate writing unit tests.

During this week, I

  • had a meeting with my mentor where we planned out the roadmap for the current week.
  • modified the babel and jest configurations to enable ES6 support in the root directory.
  • received helpful pointers from my mentor in setting up testing for visual correctness
  • wrote dummy tests in the repository
  • created a Pull Request for review

Link for the PR

My goal for the upcoming week will be to:

  1. Work on the suggested configuration changes
  2. Successfully complete Week 1 goals as mentioned in my proposal timeline
  3. Set up an external blog website
1 Like

Week 4 (June 7 to June 13)

The first goal of this week was to get new configuration settings merged into the repository. These new settings would facilitate writing React component tests using jest and @testing-library/react. The new babel settings would facilitate translation of ES6 syntax to CommonJS. With some great pointers provided by my primary mentor @sarath_ms, I was able to achieve this in the first couple of days.

The next challenge was actually writing some component tests. I had planned on covering the ‘Onboarding’ user story in the first week of this coding phase. This began with laying out a blueprint of the nodes I wanted to assert and the user interactions emulating which made the most sense. I was able to design this blueprint after taking hints from the unit and integration tests written for OONI’s other repositories.

By the end of this week, I was able to write unit tests for three NextJS components, Sections.js, QuizSteps.js and Stepper.js.

All the three components posed different challenges. Sections.js was perhaps the most straightforward, but setting up the component rendering system required writing a Higher Order JSX Component which could provide react-intl translations and a theme provider which could pass on the calculated CSS theme data upon compilation of the JSX code. Mr. Sarath provided a big hint in the form of an already existing ThemeProvider component and what prop was to be passed down with it. I was able to get tests working for Sections.js.

Stepper.js was a very visual component with no text. Writing snapshot tests for its various stages seemed like the right idea. Mr. Sarath, my primary mentor approved of this idea.

QuizSteps.js is an animation-heavy component. Animations are displayed at various stages which in turn trigger callbacks. I resorted to writing tests with timers which I figured mustn’t have been the best way to go about in dealing with animations. Mr. Sarath encouraged me to try out some APIs and I was ultimately able to figure out a better way to write tests for this particular component.

My first coding week saw some ups and downs. I’m extremely grateful to Mr. Sarath who provided frequent pointers and feedback and played a key role in solving much of my issues.

1 Like

Week 5 (June 14 to June 20)

The goal for this week was to write tests for Dashboard and Settings components. ​This involved covering Sidebar.js, DashboardHeader.js and RunTestCard.js. The original timeline involved writing tests for View Results screen as well, but I figured it’d be best to defer it to later, after the network tests are covered.

The second week of the coding phase began with identifying the components that needed to be covered for unit testing. Although I could identify a few more components which could be covered with the tests, writing the tests for them would have meant testing implementation details, which was opposed to the principles of @testing-library/react. Figuring out a way to test these couple of components will come under backlog, to be taken up in the 3rd or 4th weeks, or towards the end of the program.

Some of these tests posed multiple challenges. But I feel I was able to work around these challenges and I’m confident in these unit tests. They are fairly generic and should withstand good degree of code changes.

Apart from writing tests, I:

  • Properly set up eslint and adjusted the formatting issues with last week’s as well as this week’s PRs
  • Wrote mocks for some electron functions
  • Split up the jest configs for unit and integration tests into separate files

The PR sent for this week:

1 Like

Week 6 (June 21 to June 27)

This week’s checklist:

  • Make sure TestGroupInDetail component loads up the correct details when clicked on a particular RunTestCard
  • Website tests have an additional ‘Choose Websites’ button that directs the application to the designated route
  • Addition, Removal and Regex validation work on the Choose page

This week was particularly challenging. Writing tests for core components of the application required mocking HOCs like configuration and context providers along with several other custom functions. Writing tests for TestGroupInDetail component in particular required mocking modules like next/router as well. This week required way more research than the previous weeks. I had to look into FormatJS APIs to write custom formatting functions to test values generated from react-intlcomponents.

The other component, UrlList.js posed certain challenges as well. The original code of URL.js component produced buggy behavior in a headful environment. In a headless environment, it failed to allow the application to proceed further, thus rendering it unusable. To test this component, I had to refactor some code in the repository and modify a couple of conditions. I raised the following PR to fix this issue (it has been merged now).

I also worked on some suggestions provided by my mentor for the first week’s PR.

The master branch was updated later this week. So to avoid merge conflicts, I was suggested to rebase my last two PRs with the master branch, which I did.

The PR raised for Week 3 of coding phase is:

Week 7 (June 28 to July 4)

Week’s checklist:

  • Rebase the PRs of the past couple of weeks with the updated master branch
  • Study the codebase to understand the process of network measurements done by Probe Desktop

During this and the next week, my plan is to write comprehensive unit tests that cover the network management section of the application. This week, I resorted to writing tests for running.js component, which renders during the process of network measurement. This involved

  • Mocking log messages data and testing if they are displayed correctly
  • Correct background animations are triggered for different tests
  • The default animations are triggered for any test not present in the testlist
  • Testing if the log container is visible upon clicks.

The majority of my time this week was spent on understanding the process of network measurements than writing tests. I hope to cover the utility and event functions by the end of the next week.

The PR raised for this week :

1 Like

Week 8 (July 5 to July 11)

Week’s checklist:

  • Hold a meeting with my mentor, Mr. Sarath to seek feedback on the raised PRs and identify critical parts of the application
  • Develop a deeper understanding of the communication cycle between server, child_process and renderer
  • Mock the main and renderer IPC functions to simulate events in tests
  • Add the IPC event mock tests to running.js unit test
  • Mock modules and functions required for instantiating Ooniprobe object in jest
  • Write child_process and IPC test cases for main/utils/ooni/ooniprobe.js component
  • Write tests for main/utils/ooni/run.js
  • Write tests for main/utils/actions.js

This week was particularly challenging. My first couple of days were spent on identifying the functions and modules that required mocking and the next couple of days in trying to figure out ways to mock them. There was a lot of trial and error involved. I did manage to gain insight into concepts like sinon stubs, unix terminal signals like SIGTERM etc. I also learned about Node concepts like spawning child_process.

Covering tests for running.js and ooniprobe.js required diving much deeper into concepts of the libraries used, the testing frameworks as well as the application’s codebase. I learnt about the idiosyncrasies of Jest as a testing frameworks and its shortcomings. There were various instances when there was no easy way to customize function behaviors, so I had to resort to writing manual mock functions for every specific test case.

A couple of backlogs remain. I hope to tackle them in the upcoming weeks.

Updated PR for this week:

1 Like

Week 9 (July 12 to July 18)

What I did:

  • set up Spectron and get familiar with its APIs
  • read about WebdriverIO APIs
  • rewrote the Onboarding integration test with a couple more assertions
  • had a meeting with my mentor, Mr. Sarath, discussed changes, gathered suggestions and worked on them
  • got PRs 229 and 231 merged into the probe-desktop repositories
  • re-worked on PR 227 following a messed up rebase
  • completed the GSoC evaluation

Objectives for the upcoming week:

  • Add improvements to PR 227 and open a new PR with the changes
  • Cover network measurement in integration test. Try to find a way to mock required functions and values (like childprocess_spawn).
  • Cover i18n translation building in integration test
  • Work on main/utils/ooni/*.js backlogs if time permits
1 Like

Week 10 (July 19 to July 25)

What I did:

  • added improvements to PR 227 and got it reviewed
  • covered probe-desktop's translation building process in unit testing
  • wrote e2e tests for Dashboard section
  • wrote e2e tests for Settings sections. Wrote assertions on toggling different settings and if they had the desired effect.
  • wrote e2e tests covering individual network tests, including:
    – Complete websites test run
    – Custom websites test run
    – IM test run
    – Circumvention test run
    – Performance test run
    – Middleboxes test run

Challenges

  • Working around the limitations of spectron as a headful testing framework was a challenge. Assertion possible with spectron are more or less limited to WebdriverIO's query APIs. These APIs are only good for querying or triggering actions within the scope of renderer and not the main process of electron.

Objectives for the upcoming week

  • To refine the e2e tests currently written and add more meaningful assertions
  • Extend the e2e test suite to incorporate complete set of network test runs triggered with Dashboard’s RUN button.
  • Find a way to get generated OONI Explorer href link after a network test run.
  • Cover Test Results section of the application with e2e tests.

PRs for this week:

  1. E2E tests
    E2E tests covering Onboarding process, Dashboard and Settings pages by Eddges · Pull Request #234 · ooni/probe-desktop · GitHub

  2. Translation test
    Some tests for language translations by Eddges · Pull Request #235 · ooni/probe-desktop · GitHub

  3. Redone #227
    Week 2 tests by Eddges · Pull Request #233 · ooni/probe-desktop · GitHub

1 Like

Week 11 (July 26 to Aug 1)

What I did

  • improved PR 227, removed irrelevant test cases
  • improved runtest-individual.e2e.js. Replaced hardcoded pauses with timeout based async assertions.
  • covered the complete set of network tests triggered on clicking Dashboard’s RUN button in runtest-all.e2e.js
  • wrote a few tests for Test Results section of the Application

Challenges

  • The spectron version compatible with the electron's version is outdated. Due to this, the WebdriverIO APIs it supports are deprecated. Many webdriver selection APIs like $$ or react$ do not work at all.

Objectives for the upcoming week

  • To add more meaningful and relevant assertions to Test Results e2e tests
  • Work on backlogs for /main/utils/ooni/*.js

PRs for this week

This week’s PRs are the same as previous week’s.

1 Like

Week 12 (Aug 2 to Aug 8)

What I did

  • Adjusted test-result.e2e.js in particular to work for possible order changes in Test Results page after discussing the order with OONI community
  • Timeout values adjusted across e2e tests
  • Assertions based on array equality improved to compare sorted arrays
  • Shifted tests based on relevance between test-result.e2e.js and runtest-individual.e2e.js
  • Added over 40 screenshots for relevant e2e test cases
  • Covered Measurement results in e2e tests

Challenges

  • Understanding how the test results are displayed on the Desktop application. I found out, on discussion with my mentor that the application displays anomalous results on top. This is the reason why e2e tests were failing in CI.

Objectives for the upcoming week

  • Cover /main/utils/ooni/*.js in unit testing
  • Work on remaining TODOS as marked

Week 13 (Aug 9 to Aug 15)

What I did

  • Replaced waitFor provided by @testing-library with waitUntil, waitUntilWindowLoaded and waitUntilTextExists provided by webdriverIO across e2e tests.
  • Added OS specific Spectron test cases, covering the Autorun aspect of the application
  • Worked on mocking event-emitters in jest environment.
  • Wrote unit tests for:
    • main/actions.js
    • main/utils/config.js
    • main/utils/store.js
    • main/utils/ooni/run.js
  • Improved unit tests for main/utils/ooni/ooniprobe.js

Challenges

  • Covering the Autorun aspect of the application as it is OS dependent.
  • Improving Ooniprobe related unit tests by mocking EventEmitter. Making sure that the ooniProbe class instance catches emitted events and exhibits expected behavior.

Objectives for the upcoming week

  • Simulate event emission and catching of ‘stderr’, ‘stdout’, ‘exit’ and ‘error’ events in main/utils/ooni/ooniprobe.js, run.js and actions.js and finish up their unit tests.
  • Raise issues for the remaining TODOS to be taken up after GSoC period.
  • Documentation and Final Evaluation

PRs for this week:

Week 14 (Aug 16 to Aug 23)

What I did

  • Improved OS dependency of e2e tests, made sure e2e tests work in Windows.
  • Documented my work in a GitHub issue as suggested by one of the mentors
  • Complete the final evaluation

Find my GSoC work here: