Web Crawler¶

Description¶

Overview¶

Web Crawler is a custom application designed for the Flipper Zero device, allowing users to configure and manage HTTP requests directly from their Flipper Zero.

Requirements¶

• WiFi Dev Board or Raspberry Pi Pico W for Flipper Zero with FlipperHTTP Flash: https://github.com/jblanked/FlipperHTTP
• WiFi Access Point

Installation¶

• Download from the Official Flipper App Store: https://lab.flipper.net/apps/web_crawler
• Video tutorial: https://www.youtube.com/watch?v=lkui2Smckq4

Features¶

• Configurable Request: Specify the URL of the website you want to send a HTTP request to or download (tested up to 427Mb)
• Wi-Fi Configuration: Enter your Wi-Fi SSID and password to enable network communication.
• File Management: Automatically saves and manages received data on the device's storage, allowing users to view, rename, and delete the received data at any time.

Usage¶

1. Connection: After installing the app, turn off your Flipper, connect the WiFi Dev Board, then turn your Flipper back on.

2. Launch the Web Crawler App: Navigate to the Apps menu on your Flipper Zero, select GPIO, then scroll down and select Web Crawler.

3. Main Menu: Upon launching, you'll see a submenu containing the following options:

4. Run: Initiate the HTTP request.
5. About: View information about the Web Crawler app.

6. Settings: Set up parameters or perform file operations.

7. Settings: Select Settings and navigate to WiFi Settings. Use the Flipper Zero's navigation buttons to input and confirm your settings. Do the same for the Request settings. Once configured, these settings will be saved and used for subsequent HTTP request operations.

For testing purposes: - https://httpbin.org/get Returns GET data. - https://httpbin.org/post Returns POST data. - https://httpbin.org/put Returns PUT data. - https://httpbin.org/delete Returns DELETE data. - https://httpbin.org/bytes/1024 Returns BYTES data (DOWNLOAD method) - https://proof.ovh.net/files/1Mb.dat Returns BYTES data (DOWNLOAD method - it can download the whole file) - https://proof.ovh.net/files/10Mb.dat Returns BYTES data (DOWNLOAD method - it can download the whole file)

1. Running the Request: Select Run from the main submenu to start the HTTP request process. The app will:
2. Send Request: Transmit the HTTP request via serial to the WiFi Dev Board.
3. Receive Data: Listen for incoming data.
4. Store Data: Save the received data to the device's storage for later retrieval.

5. Log: Display detailed analysis of the operation status on the screen.

6. Accessing Received Data: After the HTTP request operation completes, you can access the received data by either:

7. Navigating to File Settings and selecting Read File (preferred method)
8. Connecting to Flipper and opening the SD/apps_data/web_crawler_app/ storage directory to access the received_data.txt file (or the file name/type customized in the settings).

Setting Up Parameters¶

1. Path (URL)

2. Enter the complete URL of the website you intend to crawl (e.g., https://www.example.com/).

3. HTTP Method

4. Choose between GET, POST, DELETE, PUT, and DOWNLOAD.

5. Headers

6. Add your required headers to be used in your HTTP requests

7. Payload

8. Type in the JSON content to be sent with your POST or PUT requests.

9. SSID (WiFi Network)

10. Provide the name of your WiFi network to enable the Flipper Zero to communicate over the network.

11. Password (WiFi Network)

12. Input the corresponding password for your WiFi network.

13. Set File Type

14. Enter your desired file extension. After saving, the app will rename your file, applying the new extension.

15. Rename File

16. Provide your desired file name. After saving, the app will rename your file with the new name.

Saving Settings¶

After entering the desired configuration parameters, the app automatically saves these settings for use during the HTTP request process. You can update these settings at any time by navigating back to the Settings menu.

Logging and Debugging¶

The Web Crawler app uses logging to help identify issues:

• Info Logs: Provide general information about the app's operations (e.g., UART initialization, sending settings).
• Error Logs: Indicate problems encountered during execution (e.g., failed to open settings file).

Connect your Flipper Zero to a computer and use a serial terminal to view these logs for detailed troubleshooting.

Known Issues¶

1. Screen Delay: Occasionally, the Run screen may get stuck on "Receiving Data".
2. If it takes longer than 10 seconds, restart your Flipper Zero.

Happy Crawling! 🕷️

Changelog¶

0.8¶

Updates from Derek Jamison: - Improved progress display. - Added connectivity check on startup.

0.7¶

• Improved memory allocation
• App restructured

0.6¶

• Added a DOWNLOAD method, which downloads the file at the specified path. This is best used for downloading images, binary files, and other non-text files.
• Increased buffer size to handle larger responses.
• Fixed the variable item lists to display as intended.

0.5 (Cleanup)¶

• Refactored using the easy flipper library
• New python library for the FlipperHTTP flash
• Removed unnecessary GPIO initialization
• Removed unnecessary logging

0.4 (New Keyboard and HTTP Methods)¶

• Updated the text input to match the text input from the UART Terminal app (big thanks to xMasterx)
• Added POST, DELETE, and PUT requests (with payloads)
• Added headers
• Added more error handling

0.3 (New Features)¶

• Updated the progress messages displayed after sending a HTTP request.
• Renamed the "Config" main submenu option to "Settings."
• Added a submenu (WiFi, File, Request) that appears when users click the "Settings" option.

Clicking WiFi displays a variable item list for SSID and Password. If you click: - SSID: A text input screen will appear with the current saved SSID pre-filled. You can change the SSID by editing the text and clicking "Save." - Password: A text input screen will appear with the current saved password pre-filled. You can change the password by editing the text and clicking "Save."

Clicking File displays a variable item list with options to Read File, Set File Type, Rename File, and Delete File. If you click: - Read File: The contents of the saved file will be displayed. - Set File Type: A text input screen will appear with the current saved file type pre-filled. You can change the file type by editing the text and clicking "Save." - Rename File: A text input screen will appear with the current saved file name pre-filled. You can change the file name by editing the text and clicking "Save." - Delete File: The saved file will be deleted.

Clicking Request displays a variable item list with a single text input option called Path. If you click Path, you can change the URL path that the GET request will be sent to.

0.2 (Stability Patch)¶

• Changed serial functions from synchronous to asynchronous.
• Added error handling for HTTP requests and WiFi Dev Board connections.
• Updated the WiFi Dev Board firmware (FlipperHTTP) to blink the green LED three times when the board is first connected, remain solid green when the board receives a serial command, and stay solid green while processing.
• Added auto-connect and auto-disconnect for WiFi when a user enters or exits the app.
• Added an option to view saved data.

0.1¶

• Initial release