Scenario / testbed
The assignment will be graded based on the groups’ ability to produce useful and correct information within the boundaries of the given time and resources.
For the experiments, we will use web servers and MONROE nodes.
Web Servers:
In order to collect web traffic measurements, we have provided you with a web server that serves TCP and QUIC:
- http://ferlin.io
Use this server to collect measurement statistics for your evaluation.
MONROE Nodes as Clients:
You will use MONROE nodes as web clients to mimic mobile users. For more information on MONROE and how to run experiments on it, please refer to the user manual: https://github.com/MONROE-PROJECT/UserManual
Below, is a brief summary of how to run an experiments on MONROE nodes:
- Go to the User Interface: https://www.monroe-system.eu/
- For this, you need to have a certificate
- Install the certificate in your browser first, then go the actual webpage listed above
- With that, you can see in the “New” tab the interface with the MONROE Scheduler
- Use this to access the nodes and schedule your experiment
- Deploying a new experiment
- Name – can be whatever you want, but make sure you can identify it
- Script – provide the name of your container in the public docket hub (misb/quic-go-client)
- Number of nodes - choose how many nodes you will use
- Explicit Node IDs: Choose among the available/provided nodes
- Additional options: Any input parameter needed for your experiment
"url": "https://www.ferlin.io","protocol": "quic", - Tick “As soon as possible”, check the availability and then deploy experiment!
- Retrieving results
- Go to the User Interface: https://www.monroe-system.eu/
- From there, check the “Status” tab
- Look for the experiment with the name you chose before, click on it and go to the “download” tab
- Example output: https://www.monroe-system.eu/user/611327/
- Script for running the experiments in MONROE, without the user interface: https://github.com/ana-cc/monroe-cli
The main purpose of the measurement is to measure the web performance with QUIC and TCP over mobile networks. We will use QUIC-GO implementation: https://github.com/lucas-clemente/quic-go. To get you up and running fast, you are provided a QUIC-GO container as a baseline under https://bitbucket.org/isabel_sanchez/quic-uio. You can update the container as you wish to get the results you desire.
For the experiment:
- On the MONROE nodes, use the following entries:
- Script – misb/quic-go-client
- Number of nodes - choose how many nodes you will use
- Explicit Node IDs: Choose among 356, 357, 358, 359, 362, 363. These nodes have Telenor, Telia Norge and ICE subscriptions. We also have Telia Mobile which is a Swedish Telia simcard roaming in Norway following the free roaming rules in Europe.
For this assignment, we assume that the target of the service provider is to deliver web traffic and that the main purpose of the measurement is to measure the flow completion time, page load time and packet loss rates for the measured traffic. Additionally you are encouraged to look for other appropriate metrics that will aid the evaluation with regard to the protocols you will be evaluating.
You may choose the tools for performing the experiments and analyse the results based on what you are familiar with and what you deem most appropriate for the purpose. The important aspect is that you are able to produce a complete and clear report at the end of the assignment period.
You should evaluate each transport protocol in competition with other flows running the same protocol and in combination with flows running the other protocol you are evaluating (TCP vs. TCP, TCP vs. QUIC and QUIC vs. QUIC).
Take into consideration that mobile networks has very dynamic behaviour that changes over time due to congestion of cells. Therefore, you need to perform enough tests to make sure that your results are representative of protocol performance and not just artifacts of random competing web traffic.
We encourage you to discuss the challenges and techniques across groups to reduce the overhead in attaining a new field of knowledge. Copying of code, scripts or experimental results, however, will be counted as cheating.
Report
You must write up the results as a technical report of no more than 4 pages in ACM format. It is expected that such a report includes the core elements presented in the lectures under “A systematic approach to performance evaluation”. The results must be based on your own experiments and your own data.
The report is evaluated by writing quality, clarity of presentation, by the trustworthiness and correctness of the results. The evaluation does not consider whether related work (citations of other papers) is included.
Evaluation details
In our evaluation of the reports, we will focus on the following elements:
- Choice of metrics, workloads, system configuration parameters and methodology for the experiments
- Use of statistical sound methods when analysing the data
- Disposition of the available time (ability to collect and present useful information within the boundaries of the available resources)
- Objectivity in defining the work, choosing metrics and workloads, in the analysis and in presenting the results
- Transparency of reporting (exposure of assumptions and limitations to the reader)
- Clarity of presentation
Bonus elements:
- Analysis of metrics, beyond the core metrics listed above, that helps illustrate the qualities of the different transport protocols.
- Analysis of experiments using other external websites to measure against (for instance Google sites).
- Analysis of the web performance when roaming (use nodes with Telia Mobile Sweden subscriptions)
- In depth analysis of web quality analysis together with metadata (e.g. RTTs, signal strength, etc…)
Formalities
The deadline for handing in your assignment is: Monday, March 19th at (15:00:00.00). Deliver your report (as PDF) at https://devilry.ifi.uio.no/.
The groups should also prepare a poster (2 x A3 pages) and a quick talk (max 5 minutes without slides) where you pitch your poster for the class on March 22nd. Name the poster with your group name, and e-mail the poster by email to inf5072@ifi.uio.no no later than noon (12:00) on March 21st. We will then print the poster for you.
For questions and course related chatter, we have created a Slack space:
https://mpglab.slack.com/messages/inf5072/
There will be a prize for best poster/presentation (awarded by an independent panel and independent of the grade).
For questions please contact: inf5072@ifi.uio.no
Resources:
Resources: