dianigionath
/
ConsultancyProject1_Auslastungsmodellierung
3
1
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Restrukturierung Repo

main
Giò 2024-07-06 19:49:36 +02:00
parent e5278442c9
commit 55cae6d83a
104 changed files with 102 additions and 12411 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

134
.gitignore vendored
View File

@ -1,115 +1,19 @@
# ---> SublimeText
# Cache files for Sublime Text
*.tmlanguage.cache
*.tmPreferences.cache
*.stTheme.cache
# Workspace files are user-specific
*.sublime-workspace
*.sqlite
# Project files should be checked into the repository, unless a significant
# proportion of contributors will probably not be using Sublime Text
# *.sublime-project
# SFTP configuration file
sftp-config.json
sftp-config-alt*.json
# Package control specific files
Package Control.last-run
Package Control.ca-list
Package Control.ca-bundle
Package Control.system-ca-bundle
Package Control.cache/
Package Control.ca-certs/
Package Control.merged-ca-bundle
Package Control.user-ca-bundle
oscrypto-ca-bundle.crt
bh_unicode_properties.cache
# Sublime-github package stores a github token in this file
# https://packagecontrol.io/packages/sublime-github
GitHub.sublime-settings
# ---> JetBrains
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
# AWS User-specific
.idea/**/aws.xml
# Generated files
.idea/**/contentModel.xml
# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
# Gradle
.idea/**/gradle.xml
.idea/**/libraries
# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/artifacts
# .idea/compiler.xml
# .idea/jarRepositories.xml
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
# *.iml
# *.ipr
scraper-python/
# CMake
cmake-build-*/
# Mongo Explorer plugin
.idea/**/mongoSettings.xml
# File-based project format
*.iws
# IntelliJ
out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# SonarLint plugin
.idea/sonarlint/
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
# Editor-based Rest Client
.idea/httpRequests
# Android studio 3.1+ serialized cache file
.idea/caches/build_file_checksums.ser
/.phpunit.cache
/node_modules
/public/build
/public/hot
/public/storage
/storage/*.key
/vendor
.env
.env.backup
.env.production
.phpunit.result.cache
Homestead.json
Homestead.yaml
auth.json
npm-debug.log
yarn-error.log
/.fleet
/.idea
/.vscode

84
README.md
View File

@ -1,2 +1,84 @@
# ConsultancyProject1_Auslastungsmodellierung
# Web Scraper e-domizil.ch
Das Repository enthält eine auf [Laravel (Version 10.x)](https://laravel.org) basierender Web Scraper für die Plattform e-domizil.ch.
## Installation
Vorbedingungen für die erfolgreiche Installation sind [Server Requirements](https://laravel.com/docs/10.x/deployment#server-requirements)
1. Das Repository klonen
```bash
git clone https://gitea.fhgr.ch/dianigionath/ConsultancyProject1_Auslastungsmodellierung.git`
```
2. Die Applikation mittels Composer installieren
```bash
php composer install
```
3. Eine Kopie der Datei .env.example nach .env erstellen und den Dateiinhalt bez. Datenbankverbindung anpassen.
```bash
cp .env.example .env`
```
Bsp. für Verbinung zu einer SQLite Datenbank.
```yaml
DB_CONNECTION=sqlite
DB_DATABASE=/absolute/path/to/database.sqlite
```
4. Mittels Artisan Console die Datenbank initialisieren
```bash
php artisan migrate
```
Erwartete Ausgabe:
```bash
WARN The SQLite database does not exist: /home/gio/database_test.sqlite.
┌ Would you like to create it? ────────────────────────────────┐
│ Yes │
└──────────────────────────────────────────────────────────────┘
INFO Preparing database.
Creating migration table ...................................................................................... 31ms DONE
INFO Running migrations.
0001_01_01_000000_create_users_table .......................................................................... 57ms DONE
0001_01_01_000001_create_cache_table .......................................................................... 18ms DONE
2019_12_14_000001_create_personal_access_tokens_table ......................................................... 36ms DONE
2024_03_15_142227_create_regions_table ........................................................................ 10ms DONE
2024_03_15_142228_create_seeds_table .......................................................................... 18ms DONE
2024_03_15_142257_create_properties_table ..................................................................... 17ms DONE
2024_03_15_142550_create_extractions_table .................................................................... 10ms DONE
2024_03_15_142625_create_exceptions_table ..................................................................... 10ms DONE
2024_03_15_162023_create_jobs_table ........................................................................... 18ms DONE
2024_04_08_115153_create_failed_jobs_table .................................................................... 32ms DONE
```
5. Gewünschte Region(en) mittels Artisan Konsole hinzufügen:
```bash
php artisan scraper:add-region
```
Mögliche erwartete Ausgabe:
```bash
Type in desired region:
> Davos
Choose desired region:
[5460aea91d044] Davos
[5390628eeaa24] Davos Davos Platz
[5460adf3d7913] Davos Clavadel
[565847a969c59] Prättigau/Davos
[5460adf87857d] Davos Wolfgang
[5460adf8f3e46] Davos Monstein
> Davos
New Region created {"name":"Davos","updated_at":"2024-07-06T17:24:09.000000Z","created_at":"2024-07-06T17:24:09.000000Z","id":1}
New Seed added {"uri":"https:\/\/www.e-domizil.ch\/search\/5460aea91d044?_format=json","region_id":1,"updated_at":"2024-07-06T17:24:09.000000Z","created_at":"2024-07-06T17:24:09.000000Z","id":1}
```
6. Zum Schluss sind Cronjobs einzurichten, welche den Webscraper regelmässig ausführt
Alle drei Tage um 02:00 die Scraping Jobs erstellen:
```bash
0 2 */3 * * /usr/local/bin/php ConsultancyProject1_Auslastungsmodellierung/artisan scrape:jobs
```
Jeden Tag alle drei Stunden zwischen 04:00 bis 23:00 mit einer zufälligen Verzögerung bis zu einer Stunde den Queue Worker für das Abarbeiten von 250 Jobs ausführen.
```bash
0 4,7,9,11,13,15,17,19,21,23 * * * sleep $((RANDOM \% 60))m ; /absolute/path/to/bin/php /absolute/path/to/artisan queue:work --max-jobs=250 --stop-when-empty --max-time=7200
```

44
docs/schema.dbml
View File

@ -1,44 +0,0 @@
// Use DBML to define your database structure
// Docs: https://dbml.dbdiagram.io/docs
Table seeds [note: 'Table contains the URIs which are used for the initial scraping.'] {
seed_id integer [primary key]
uri text [not null, unique]
region_id integer [not null, ref: > regions.region_id]
}
Table regions {
region_id integer [primary key]
name varchar(255) [not null]
}
Table properties {
property_id integer [primary key]
property_platform_id varchar(255) [unique, not null, note: 'uuid from platform beeing used']
seed_id integer [not null, ref: > seeds.seed_id]
check_data json [note: 'for storing data, which is beeing used for consistency checks. E. g. geo_dates or title']
last_found timestamp
created_at timestamp
}
Table extractions {
extraction_id integer [primary key]
property_id integer [unique, ref: > properties.property_id]
body text [not null]
header text [not null]
type types [not null]
created_at timestamp [not null]
}
enum types {
property
calendar
offer
}
Table exceptions {
exception_id integer [primary key]
exception json [not null, note: "exception while scraping (e. g. HTTP error message) and called url."]
type types [not null]
property_id integer [not null, ref: > properties.property_id, note: "either a property_id"]
}

204
docs/schema.svg
View File

@ -1,204 +0,0 @@
<svg width="100%" height="100%" viewBox="0.00 0.00 2466.42 1017.92" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" style="width: 100%; height: 100%; max-height: 1018pt; max-width: 2466pt;">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 1013.92)">
<title>dbml</title>
<!-- enteties -->
<g id="enteties" class="node">
<title>enteties</title>
<ellipse fill="none" stroke="black" stroke-width="0" cx="1004.74" cy="-130.11" rx="185.02" ry="130.22"></ellipse>
<polygon fill="#29235c" stroke="transparent" points="875.74,-160.11 875.74,-220.11 1133.74,-220.11 1133.74,-160.11 875.74,-160.11"></polygon>
<polygon fill="none" stroke="#29235c" points="875.74,-160.11 875.74,-220.11 1133.74,-220.11 1133.74,-160.11 875.74,-160.11"></polygon>
<text text-anchor="start" x="886.47" y="-181.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#ffffff"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;enteties &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</text>
<polygon fill="#e7e2dd" stroke="transparent" points="875.74,-100.11 875.74,-160.11 1133.74,-160.11 1133.74,-100.11 875.74,-100.11"></polygon>
<polygon fill="none" stroke="#29235c" points="875.74,-100.11 875.74,-160.11 1133.74,-160.11 1133.74,-100.11 875.74,-100.11"></polygon>
<text text-anchor="start" x="910.49" y="-121.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#1d71b8"> &nbsp;&nbsp;&nbsp;property &nbsp;&nbsp;&nbsp;</text>
<polygon fill="#e7e2dd" stroke="transparent" points="875.74,-40.11 875.74,-100.11 1133.74,-100.11 1133.74,-40.11 875.74,-40.11"></polygon>
<polygon fill="none" stroke="#29235c" points="875.74,-40.11 875.74,-100.11 1133.74,-100.11 1133.74,-40.11 875.74,-40.11"></polygon>
<text text-anchor="start" x="892.69" y="-61.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#1d71b8"> &nbsp;&nbsp;&nbsp;occupancy &nbsp;&nbsp;&nbsp;</text>
<polygon fill="none" stroke="#29235c" stroke-width="2" points="874.74,-39.11 874.74,-221.11 1134.74,-221.11 1134.74,-39.11 874.74,-39.11"></polygon>
</g>
<!-- seeds -->
<g id="seeds" class="node">
<title>seeds</title>
<ellipse fill="none" stroke="black" stroke-width="0" cx="1679.25" cy="-800.11" rx="232.78" ry="172.57"></ellipse>
<polygon fill="#1d71b8" stroke="transparent" points="1517.25,-860.11 1517.25,-920.11 1842.25,-920.11 1842.25,-860.11 1517.25,-860.11"></polygon>
<polygon fill="none" stroke="#29235c" points="1517.25,-860.11 1517.25,-920.11 1842.25,-920.11 1842.25,-860.11 1517.25,-860.11"></polygon>
<text text-anchor="start" x="1574.82" y="-881.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#ffffff"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;seeds &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</text>
<polygon fill="#e7e2dd" stroke="transparent" points="1517.25,-800.11 1517.25,-860.11 1842.25,-860.11 1842.25,-800.11 1517.25,-800.11"></polygon>
<polygon fill="none" stroke="#29235c" points="1517.25,-800.11 1517.25,-860.11 1842.25,-860.11 1842.25,-800.11 1517.25,-800.11"></polygon>
<text text-anchor="start" x="1528.25" y="-821.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">seed_id</text>
<text text-anchor="start" x="1640.3" y="-821.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1733.45" y="-821.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<polygon fill="#e7e2dd" stroke="transparent" points="1517.25,-740.11 1517.25,-800.11 1842.25,-800.11 1842.25,-740.11 1517.25,-740.11"></polygon>
<polygon fill="none" stroke="#29235c" points="1517.25,-740.11 1517.25,-800.11 1842.25,-800.11 1842.25,-740.11 1517.25,-740.11"></polygon>
<text text-anchor="start" x="1528.25" y="-760.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">uri &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1740.59" y="-761.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">text</text>
<text text-anchor="start" x="1792.16" y="-761.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="1801.06" y="-761.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="#e7e2dd" stroke="transparent" points="1517.25,-680.11 1517.25,-740.11 1842.25,-740.11 1842.25,-680.11 1517.25,-680.11"></polygon>
<polygon fill="none" stroke="#29235c" points="1517.25,-680.11 1517.25,-740.11 1842.25,-740.11 1842.25,-680.11 1517.25,-680.11"></polygon>
<text text-anchor="start" x="1528.17" y="-700.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">region_id &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1694.81" y="-701.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<text text-anchor="start" x="1792.61" y="-701.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="1801.5" y="-701.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="none" stroke="#29235c" stroke-width="2" points="1515.75,-679.11 1515.75,-921.11 1842.75,-921.11 1842.75,-679.11 1515.75,-679.11"></polygon>
</g>
<!-- regions -->
<g id="regions" class="node">
<title>regions</title>
<ellipse fill="none" stroke="black" stroke-width="0" cx="2203.16" cy="-830.11" rx="255.03" ry="130.22"></ellipse>
<polygon fill="#1d71b8" stroke="transparent" points="2025.16,-860.11 2025.16,-920.11 2382.16,-920.11 2382.16,-860.11 2025.16,-860.11"></polygon>
<polygon fill="none" stroke="#29235c" points="2025.16,-860.11 2025.16,-920.11 2382.16,-920.11 2382.16,-860.11 2025.16,-860.11"></polygon>
<text text-anchor="start" x="2088.95" y="-881.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#ffffff"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;regions &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</text>
<polygon fill="#e7e2dd" stroke="transparent" points="2025.16,-800.11 2025.16,-860.11 2382.16,-860.11 2382.16,-800.11 2025.16,-800.11"></polygon>
<polygon fill="none" stroke="#29235c" points="2025.16,-800.11 2025.16,-860.11 2382.16,-860.11 2382.16,-800.11 2025.16,-800.11"></polygon>
<text text-anchor="start" x="2036.16" y="-821.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">region_id</text>
<text text-anchor="start" x="2167.74" y="-821.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="2273.36" y="-821.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<polygon fill="#e7e2dd" stroke="transparent" points="2025.16,-740.11 2025.16,-800.11 2382.16,-800.11 2382.16,-740.11 2025.16,-740.11"></polygon>
<polygon fill="none" stroke="#29235c" points="2025.16,-740.11 2025.16,-800.11 2382.16,-800.11 2382.16,-740.11 2025.16,-740.11"></polygon>
<text text-anchor="start" x="2035.86" y="-760.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">name &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="2150.94" y="-761.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">varchar(255)</text>
<text text-anchor="start" x="2332.28" y="-761.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="2341.18" y="-761.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="none" stroke="#29235c" stroke-width="2" points="2023.66,-739.11 2023.66,-921.11 2382.66,-921.11 2382.66,-739.11 2023.66,-739.11"></polygon>
</g>
<!-- seeds&#45;&gt;regions -->
<!-- seeds&#45;&gt;regions -->
<g id="edge2" class="edge">
<title>seeds:e-&gt;regions:w</title>
<path fill="none" stroke="#29235c" stroke-width="3" d="M1843.25,-710.11C1936.25,-710.11 1928.4,-821.59 2014.11,-829.65"></path>
<polygon fill="#29235c" stroke="#29235c" stroke-width="3" points="2014.01,-833.15 2024.16,-830.11 2014.33,-826.15 2014.01,-833.15"></polygon>
<text text-anchor="middle" x="2015.26" y="-839.71" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">1</text>
<text text-anchor="middle" x="1837.03" y="-719.71" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">*</text>
</g>
<!-- properties -->
<g id="properties" class="node">
<title>properties</title>
<ellipse fill="none" stroke="black" stroke-width="0" cx="1004.74" cy="-710.11" rx="405.76" ry="299.63"></ellipse>
<polygon fill="#1d71b8" stroke="transparent" points="719.74,-860.11 719.74,-920.11 1289.74,-920.11 1289.74,-860.11 719.74,-860.11"></polygon>
<polygon fill="none" stroke="#29235c" points="719.74,-860.11 719.74,-920.11 1289.74,-920.11 1289.74,-860.11 719.74,-860.11"></polygon>
<text text-anchor="start" x="871.37" y="-881.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#ffffff"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;properties &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</text>
<polygon fill="#e7e2dd" stroke="transparent" points="719.74,-800.11 719.74,-860.11 1289.74,-860.11 1289.74,-800.11 719.74,-800.11"></polygon>
<polygon fill="none" stroke="#29235c" points="719.74,-800.11 719.74,-860.11 1289.74,-860.11 1289.74,-800.11 719.74,-800.11"></polygon>
<text text-anchor="start" x="730.74" y="-821.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">property_id</text>
<text text-anchor="start" x="890.77" y="-821.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1180.93" y="-821.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<polygon fill="#e7e2dd" stroke="transparent" points="719.74,-740.11 719.74,-800.11 1289.74,-800.11 1289.74,-740.11 719.74,-740.11"></polygon>
<polygon fill="none" stroke="#29235c" points="719.74,-740.11 719.74,-800.11 1289.74,-800.11 1289.74,-740.11 719.74,-740.11"></polygon>
<text text-anchor="start" x="730.26" y="-760.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">property_platform_id &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1058.52" y="-761.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">varchar(255)</text>
<text text-anchor="start" x="1239.86" y="-761.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="1248.76" y="-761.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="#e7e2dd" stroke="transparent" points="719.74,-680.11 719.74,-740.11 1289.74,-740.11 1289.74,-680.11 719.74,-680.11"></polygon>
<polygon fill="none" stroke="#29235c" points="719.74,-680.11 719.74,-740.11 1289.74,-740.11 1289.74,-680.11 719.74,-680.11"></polygon>
<text text-anchor="start" x="730.74" y="-700.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">seed_id &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1141.84" y="-701.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<text text-anchor="start" x="1239.65" y="-701.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="1248.54" y="-701.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="#e7e2dd" stroke="transparent" points="719.74,-620.11 719.74,-680.11 1289.74,-680.11 1289.74,-620.11 719.74,-620.11"></polygon>
<polygon fill="none" stroke="#29235c" points="719.74,-620.11 719.74,-680.11 1289.74,-680.11 1289.74,-620.11 719.74,-620.11"></polygon>
<text text-anchor="start" x="730.74" y="-640.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">check_data &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1220.06" y="-641.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">json</text>
<polygon fill="#e7e2dd" stroke="transparent" points="719.74,-560.11 719.74,-620.11 1289.74,-620.11 1289.74,-560.11 719.74,-560.11"></polygon>
<polygon fill="none" stroke="#29235c" points="719.74,-560.11 719.74,-620.11 1289.74,-620.11 1289.74,-560.11 719.74,-560.11"></polygon>
<text text-anchor="start" x="730.74" y="-580.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">last_found &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1131.18" y="-581.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">timestamp</text>
<polygon fill="#e7e2dd" stroke="transparent" points="719.74,-500.11 719.74,-560.11 1289.74,-560.11 1289.74,-500.11 719.74,-500.11"></polygon>
<polygon fill="none" stroke="#29235c" points="719.74,-500.11 719.74,-560.11 1289.74,-560.11 1289.74,-500.11 719.74,-500.11"></polygon>
<text text-anchor="start" x="730.74" y="-520.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">created_at &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="1131.18" y="-521.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">timestamp</text>
<polygon fill="none" stroke="#29235c" stroke-width="2" points="718.74,-499.11 718.74,-921.11 1290.74,-921.11 1290.74,-499.11 718.74,-499.11"></polygon>
</g>
<!-- properties&#45;&gt;seeds -->
<!-- properties&#45;&gt;seeds -->
<g id="edge4" class="edge">
<title>properties:e-&gt;seeds:w</title>
<path fill="none" stroke="#29235c" stroke-width="3" d="M1290.74,-710.11C1400.84,-710.11 1402.71,-822.95 1506.25,-829.78"></path>
<polygon fill="#29235c" stroke="#29235c" stroke-width="3" points="1506.15,-833.28 1516.25,-830.11 1506.37,-826.29 1506.15,-833.28"></polygon>
<text text-anchor="middle" x="1507.36" y="-839.71" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">1</text>
<text text-anchor="middle" x="1296.96" y="-719.71" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">*</text>
</g>
<!-- occupancies -->
<g id="occupancies" class="node">
<title>occupancies</title>
<ellipse fill="none" stroke="black" stroke-width="0" cx="281.43" cy="-740.11" rx="281.36" ry="257.27"></ellipse>
<polygon fill="#1d71b8" stroke="transparent" points="84.43,-860.11 84.43,-920.11 478.43,-920.11 478.43,-860.11 84.43,-860.11"></polygon>
<polygon fill="none" stroke="#29235c" points="84.43,-860.11 84.43,-920.11 478.43,-920.11 478.43,-860.11 84.43,-860.11"></polygon>
<text text-anchor="start" x="130.26" y="-881.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#ffffff"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;occupancies &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</text>
<polygon fill="#e7e2dd" stroke="transparent" points="84.43,-800.11 84.43,-860.11 478.43,-860.11 478.43,-800.11 84.43,-800.11"></polygon>
<polygon fill="none" stroke="#29235c" points="84.43,-800.11 84.43,-860.11 478.43,-860.11 478.43,-800.11 84.43,-800.11"></polygon>
<text text-anchor="start" x="95.43" y="-821.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">occupancy_id</text>
<text text-anchor="start" x="291.05" y="-821.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="369.63" y="-821.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<polygon fill="#e7e2dd" stroke="transparent" points="84.43,-740.11 84.43,-800.11 478.43,-800.11 478.43,-740.11 84.43,-740.11"></polygon>
<polygon fill="none" stroke="#29235c" points="84.43,-740.11 84.43,-800.11 478.43,-800.11 478.43,-740.11 84.43,-740.11"></polygon>
<text text-anchor="start" x="95.43" y="-760.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">property_id &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="369.63" y="-761.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<polygon fill="#e7e2dd" stroke="transparent" points="84.43,-680.11 84.43,-740.11 478.43,-740.11 478.43,-680.11 84.43,-680.11"></polygon>
<polygon fill="none" stroke="#29235c" points="84.43,-680.11 84.43,-740.11 478.43,-740.11 478.43,-680.11 84.43,-680.11"></polygon>
<text text-anchor="start" x="95.43" y="-700.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">occupancy &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="369.66" y="-701.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">json</text>
<text text-anchor="start" x="428.34" y="-701.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="437.23" y="-701.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="#e7e2dd" stroke="transparent" points="84.43,-620.11 84.43,-680.11 478.43,-680.11 478.43,-620.11 84.43,-620.11"></polygon>
<polygon fill="none" stroke="#29235c" points="84.43,-620.11 84.43,-680.11 478.43,-680.11 478.43,-620.11 84.43,-620.11"></polygon>
<text text-anchor="start" x="95.43" y="-640.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">header &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="376.76" y="-641.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">text</text>
<text text-anchor="start" x="428.34" y="-641.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="437.23" y="-641.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="#e7e2dd" stroke="transparent" points="84.43,-560.11 84.43,-620.11 478.43,-620.11 478.43,-560.11 84.43,-560.11"></polygon>
<polygon fill="none" stroke="#29235c" points="84.43,-560.11 84.43,-620.11 478.43,-620.11 478.43,-560.11 84.43,-560.11"></polygon>
<text text-anchor="start" x="95.06" y="-580.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">created_at &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="281.1" y="-581.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">timestamp</text>
<text text-anchor="start" x="428.66" y="-581.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="437.55" y="-581.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="none" stroke="#29235c" stroke-width="2" points="83.43,-559.11 83.43,-921.11 479.43,-921.11 479.43,-559.11 83.43,-559.11"></polygon>
</g>
<!-- occupancies&#45;&gt;properties -->
<!-- occupancies&#45;&gt;properties -->
<g id="edge6" class="edge">
<title>occupancies:e-&gt;properties:w</title>
<path fill="none" stroke="#29235c" stroke-width="3" d="M479.43,-770.11C585.65,-770.11 607.75,-826.42 708.71,-829.94"></path>
<polygon fill="#29235c" stroke="#29235c" stroke-width="3" points="708.68,-833.44 718.74,-830.11 708.8,-826.44 708.68,-833.44"></polygon>
<text text-anchor="middle" x="709.84" y="-839.71" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">1</text>
<text text-anchor="middle" x="485.65" y="-779.71" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">*</text>
</g>
<!-- exceptions -->
<g id="exceptions" class="node">
<title>exceptions</title>
<ellipse fill="none" stroke="black" stroke-width="0" cx="281.43" cy="-250.11" rx="239.92" ry="214.92"></ellipse>
<polygon fill="#1d71b8" stroke="transparent" points="114.43,-340.11 114.43,-400.11 449.43,-400.11 449.43,-340.11 114.43,-340.11"></polygon>
<polygon fill="none" stroke="#29235c" points="114.43,-340.11 114.43,-400.11 449.43,-400.11 449.43,-340.11 114.43,-340.11"></polygon>
<text text-anchor="start" x="143.21" y="-361.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#ffffff"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;exceptions &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</text>
<polygon fill="#e7e2dd" stroke="transparent" points="114.43,-280.11 114.43,-340.11 449.43,-340.11 449.43,-280.11 114.43,-280.11"></polygon>
<polygon fill="none" stroke="#29235c" points="114.43,-280.11 114.43,-340.11 449.43,-340.11 449.43,-280.11 114.43,-280.11"></polygon>
<text text-anchor="start" x="124.96" y="-301.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">extraction_id</text>
<text text-anchor="start" x="306.33" y="-301.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="341.03" y="-301.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<polygon fill="#e7e2dd" stroke="transparent" points="114.43,-220.11 114.43,-280.11 449.43,-280.11 449.43,-220.11 114.43,-220.11"></polygon>
<polygon fill="none" stroke="#29235c" points="114.43,-220.11 114.43,-280.11 449.43,-280.11 449.43,-220.11 114.43,-220.11"></polygon>
<text text-anchor="start" x="125.43" y="-240.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">exception &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="340.66" y="-241.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">json</text>
<text text-anchor="start" x="399.34" y="-241.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="408.23" y="-241.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="#e7e2dd" stroke="transparent" points="114.43,-160.11 114.43,-220.11 449.43,-220.11 449.43,-160.11 114.43,-160.11"></polygon>
<polygon fill="none" stroke="#29235c" points="114.43,-160.11 114.43,-220.11 449.43,-220.11 449.43,-160.11 114.43,-160.11"></polygon>
<text text-anchor="start" x="125.43" y="-180.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">entity &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="287.3" y="-181.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">enteties</text>
<text text-anchor="start" x="399.34" y="-181.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="408.23" y="-181.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="#e7e2dd" stroke="transparent" points="114.43,-100.11 114.43,-160.11 449.43,-160.11 449.43,-100.11 114.43,-100.11"></polygon>
<polygon fill="none" stroke="#29235c" points="114.43,-100.11 114.43,-160.11 449.43,-160.11 449.43,-100.11 114.43,-100.11"></polygon>
<text text-anchor="start" x="125.43" y="-120.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c">entity_id &nbsp;&nbsp;&nbsp;</text>
<text text-anchor="start" x="301.54" y="-121.31" font-family="Helvetica,sans-Serif" font-style="italic" font-size="32.00" fill="#29235c">integer</text>
<text text-anchor="start" x="399.34" y="-121.31" font-family="Helvetica,sans-Serif" font-size="32.00" fill="#29235c"> </text>
<text text-anchor="start" x="408.23" y="-121.31" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="32.00" fill="#29235c">(!)</text>
<polygon fill="none" stroke="#29235c" stroke-width="2" points="112.93,-99.11 112.93,-401.11 449.93,-401.11 449.93,-99.11 112.93,-99.11"></polygon>
</g>
<!-- exceptions&#45;&gt;enteties -->
<g id="edge7" class="edge">
<title>exceptions:e-&gt;enteties:w</title>
<path fill="none" stroke="#29235c" stroke-width="3" d="M450.43,-190.11C639.01,-190.11 686.16,-190.11 874.74,-190.11"></path>
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 23 KiB

7
docs/seeds.csv
View File

@ -1,7 +0,0 @@
URI,description
https://www.e-domizil.ch/search/632d3fb65adbe?_format=json&adults=1&duration=7,Heidiland
https://www.e-domizil.ch/search/5460aea91d044?_format=json&adults=1&duration=7,Davos (GR)
https://www.e-domizil.ch/search/5555b19e174fc?_format=json&adults=1&duration=7,Engadin
https://www.e-domizil.ch/search/5460aea9284b5?_format=json&adults=1&duration=7,St. Moritz
1 URI description
2 https://www.e-domizil.ch/search/632d3fb65adbe?_format=json&adults=1&duration=7 Heidiland
3 https://www.e-domizil.ch/search/5460aea91d044?_format=json&adults=1&duration=7 Davos (GR)
4 https://www.e-domizil.ch/search/5555b19e174fc?_format=json&adults=1&duration=7 Engadin
5 https://www.e-domizil.ch/search/5460aea9284b5?_format=json&adults=1&duration=7 St. Moritz

4
docs/table-export.sql
View File

@ -1,4 +0,0 @@
SELECT CONCAT('https://www.e-domizil.ch/rental/offer/', properties.property_platform_id) AS url, extractions.body, extractions.header, extractions.type, extractions.created_at, regions.name FROM `extractions`
LEFT JOIN properties ON properties.id = extractions.property_id
LEFT JOIN seeds ON seeds.id = properties.seed_id
LEFT JOIN regions ON regions.id = seeds.region_id;

18
scraper/.editorconfig
View File

@ -1,18 +0,0 @@
root = true
[*]
charset = utf-8
end_of_line = lf
indent_size = 4
indent_style = space
insert_final_newline = true
trim_trailing_whitespace = true
[*.md]
trim_trailing_whitespace = false
[*.{yml,yaml}]
indent_size = 2
[docker-compose.yml]
indent_size = 4

28
scraper/.env.example
View File

@ -1,28 +0,0 @@
APP_NAME=Laravel
APP_ENV=local
APP_KEY=
APP_DEBUG=true
APP_URL=http://localhost
LOG_CHANNEL=stack
LOG_DEPRECATIONS_CHANNEL=null
LOG_LEVEL=debug
# mysql or mariadb connection
# DB_CONNECTION=mysql
# DB_HOST=127.0.0.1
# DB_PORT=3306
# DB_DATABASE=laravel
# DB_USERNAME=root
# DB_PASSWORD=
# sqlite database
DB_CONNECTION=sqlite
DB_DATABASE=/absolute/path/to/database.sqlite
BROADCAST_DRIVER=log
CACHE_DRIVER=file
FILESYSTEM_DISK=local
QUEUE_CONNECTION=database
SESSION_DRIVER=file
SESSION_LIFETIME=120

11
scraper/.gitattributes vendored
View File

@ -1,11 +0,0 @@
* text=auto eol=lf
*.blade.php diff=html
*.css diff=css
*.html diff=html
*.md diff=markdown
*.php diff=php
/.github export-ignore
CHANGELOG.md export-ignore
.styleci.yml export-ignore

19
scraper/.gitignore vendored
View File

@ -1,19 +0,0 @@
/.phpunit.cache
/node_modules
/public/build
/public/hot
/public/storage
/storage/*.key
/vendor
.env
.env.backup
.env.production
.phpunit.result.cache
Homestead.json
Homestead.yaml
auth.json
npm-debug.log
yarn-error.log
/.fleet
/.idea
/.vscode

84
scraper/README.md
View File

@ -1,84 +0,0 @@
# Web Scraper e-domizil.ch
Das Repository enthält eine auf [Laravel (Version 10.x)](https://laravel.org) basierender Web Scraper für die Plattform e-domizil.ch.
## Installation
Vorbedingungen für die erfolgreiche Installation sind [Server Requirements](https://laravel.com/docs/10.x/deployment#server-requirements)
1. Das Repository klonen
```bash
git clone https://gitea.fhgr.ch/dianigionath/ConsultancyProject1_Auslastungsmodellierung.git`
```
2. Die Applikation mittels Composer installieren
```bash
php composer install
```
3. Eine Kopie der Datei .env.example nach .env erstellen und den Dateiinhalt bez. Datenbankverbindung anpassen.
```bash
cp .env.example .env`
```
Bsp. für Verbinung zu einer SQLite Datenbank.
```yaml
DB_CONNECTION=sqlite
DB_DATABASE=/absolute/path/to/database.sqlite
```
4. Mittels Artisan Console die Datenbank initialisieren
```bash
php artisan migrate
```
Erwartete Ausgabe:
```bash
WARN The SQLite database does not exist: /home/gio/database_test.sqlite.
┌ Would you like to create it? ────────────────────────────────┐
│ Yes │
└──────────────────────────────────────────────────────────────┘
INFO Preparing database.
Creating migration table .......................................................................... 31ms DONE
INFO Running migrations.
0001_01_01_000000_create_users_table .......................................................................... 57ms DONE
0001_01_01_000001_create_cache_table .......................................................................... 18ms DONE
2019_12_14_000001_create_personal_access_tokens_table .......................................................................... 36ms DONE
2024_03_15_142227_create_regions_table .......................................................................... 10ms DONE
2024_03_15_142228_create_seeds_table .......................................................................... 18ms DONE
2024_03_15_142257_create_properties_table .......................................................................... 17ms DONE
2024_03_15_142550_create_extractions_table .......................................................................... 10ms DONE
2024_03_15_142625_create_exceptions_table .......................................................................... 10ms DONE
2024_03_15_162023_create_jobs_table .......................................................................... 18ms DONE
2024_04_08_115153_create_failed_jobs_table .......................................................................... 32ms DONE
```
5. Gewünschte Region(en) mittels Artisan Konsole hinzufügen:
```bash
php artisan scraper:add-region
```
Mögliche erwartete Ausgabe:
```bash
Type in desired region:
> Davos
Choose desired region:
[5460aea91d044] Davos
[5390628eeaa24] Davos Davos Platz
[5460adf3d7913] Davos Clavadel
[565847a969c59] Prättigau/Davos
[5460adf87857d] Davos Wolfgang
[5460adf8f3e46] Davos Monstein
> Davos
New Region created {"name":"Davos","updated_at":"2024-07-06T17:24:09.000000Z","created_at":"2024-07-06T17:24:09.000000Z","id":1}
New Seed added {"uri":"https:\/\/www.e-domizil.ch\/search\/5460aea91d044?_format=json","region_id":1,"updated_at":"2024-07-06T17:24:09.000000Z","created_at":"2024-07-06T17:24:09.000000Z","id":1}
```
6. Zum Schluss sind Cronjobs einzurichten, welche den Webscraper regelmässig ausführt
Alle drei Tage um 02:00 die Scraping Jobs erstellen:
```bash
0 2 */3 * * /usr/local/bin/php ConsultancyProject1_Auslastungsmodellierung/artisan scrape:jobs
```
Jeden Tag alle drei Stunden zwischen 04:00 bis 23:00 mit einer zufälligen Verzögerung bis zu einer Stunde den Queue Worker für das Abarbeiten von 250 Jobs ausführen.
```bash
0 4,7,9,11,13,15,17,19,21,23 * * * sleep $((RANDOM \% 60))m ; /absolute/path/to/bin/php /absolute/path/to/artisan queue:work --max-jobs=250 --stop-when-empty --max-time=7200
```

76
scraper/app/Console/Commands/scraperAddRegion.php
View File

@ -1,76 +0,0 @@
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Http;
use App\Models\Regions;
use App\Models\Seed;
class scraperAddRegion extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'scraper:add-region';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Add new region to database.';
/**
* Execute the console command.
*/
public function handle()
{
$name = $this->ask('Type in desired region');
$suggestions = Http::get('https://www.e-domizil.ch/api/v2/autocomplete?q='.$name.'&limit=6');
$options = [];
if($suggestions->successful()){
$suggestionsArr = json_decode($suggestions->body(), 1);
if(count($suggestionsArr['suggestions']) > 0){
foreach ($suggestionsArr['suggestions'] as $suggestion) {
$options[$suggestion['id']] = $suggestion['fullTitle'];
}
$choice = $this->choice(
'Choose desired region',
$options
);
$id = $choice;
$name = $options[$choice];
}
}
$region = Regions::create([
'name' => $name
]);
$this->info('New Region created '.$region);
if(count($options) > 0){
$seed = Seed::create([
'uri' => 'https://www.e-domizil.ch/search/'.$id.'?_format=json',
'region_id' => $region['id']
]);
$this->info('New Seed added '.$seed);
}
}
}

55
scraper/app/Console/Commands/scraperAddSeed.php
View File

@ -1,55 +0,0 @@
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use App\Models\Regions;
use App\Models\Seed;
class scraperAddSeed extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'scraper:add-seed';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Add new seed to database.';
/**
* Execute the console command.
*/
public function handle()
{
$regions = Regions::all()->pluck('name', 'id')->all();
$regionChoice = $this->choice(
'For which region do you want to add the seed?',
$regions
);
$url = $this->ask('Please input the seed url');
if(!str_contains($url, 'e-domizil.ch')){
$this->error('Given seed url is not valid.');
}else{
$seed = Seed::create([
'uri' => $url,
'region_id' => array_search($regionChoice, $regions)
]);
$this->info('New Seed has been added: '.$seed);
}
}
}

33
scraper/app/Console/Commands/scraperCreateJobs.php
View File

@ -1,33 +0,0 @@
<?php
namespace App\Console\Commands;
use App\Scraper\Edomizil;
use App\Models\Property;
use Illuminate\Console\Command;
class scraperCreateJobs extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'scraper:jobs';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Create and dispatch jobs for scraping';
/**
* Execute the console command.
*/
public function handle()
{
Edomizil::dispatchPropertyJobs();
Edomizil::dispatchPropertyDataJobs();
}
}

40
scraper/app/Console/Commands/scraperProperty.php
View File

@ -1,40 +0,0 @@
<?php
namespace App\Console\Commands;
use App\Scraper\Edomizil;
use Illuminate\Console\Command;
class scraperProperty extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'scraper:scrape-properties';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Scrape for properties.';
/**
* Execute the console command.
*/
public function handle()
{
$seeds = Edomizil::getAllSeeds();
foreach($seeds as $seed){
$this->info('Scraping '.$seed);
$scraper = Edomizil::ScrapeProperty($seed);
$this->info($scraper.' properties were found.');
}
$this->info('Finished scraping.');
}
}

50
scraper/app/Console/Commands/scraperPropertyDate.php
View File

@ -1,50 +0,0 @@
<?php
namespace App\Console\Commands;
use App\Scraper\Edomizil;
use App\Models\Property;
use Illuminate\Console\Command;
class scraperPropertyDate extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'scraper:scrape-propertydata {propertyId?} {--seed=}';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Scrapes the data from one random property or property with ID from argument.';
/**
* Execute the console command.
*/
public function handle()
{
$propertyId = $this->argument('propertyId');
$seed = $this->option('seed');
if($seed){
$property = Property::select('id','property_platform_id')->where('seed_id', $seed)->inRandomOrder()->take(1)->get();
} else {
if($propertyId && is_numeric($propertyId)){
$property = Property::select('id','property_platform_id')->where('id', 'like', $propertyId)->get();
}else{
$property = Property::select('id','property_platform_id')->inRandomOrder()->take(1)->get();
}
}
$data = Edomizil::scrapePropertyData($property[0]);
$this->info('Scraping for property '.$property[0]->property_platform_id.' has ended. Here is the result: '.$data);
}
}

27
scraper/app/Console/Kernel.php
View File

@ -1,27 +0,0 @@
<?php
namespace App\Console;
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;
class Kernel extends ConsoleKernel
{
/**
* Define the application's command schedule.
*/
protected function schedule(Schedule $schedule): void
{
// $schedule->command('inspire')->hourly();
}
/**
* Register the commands for the application.
*/
protected function commands(): void
{
$this->load(__DIR__.'/Commands');
require base_path('routes/console.php');
}
}

30
scraper/app/Exceptions/Handler.php
View File

@ -1,30 +0,0 @@
<?php
namespace App\Exceptions;
use Illuminate\Foundation\Exceptions\Handler as ExceptionHandler;
use Throwable;
class Handler extends ExceptionHandler
{
/**
* The list of the inputs that are never flashed to the session on validation exceptions.
*
* @var array<int, string>
*/
protected $dontFlash = [
'current_password',
'password',
'password_confirmation',
];
/**
* Register the exception handling callbacks for the application.
*/
public function register(): void
{
$this->reportable(function (Throwable $e) {
//
});
}
}

12
scraper/app/Http/Controllers/Controller.php
View File

@ -1,12 +0,0 @@
<?php
namespace App\Http\Controllers;
use Illuminate\Foundation\Auth\Access\AuthorizesRequests;
use Illuminate\Foundation\Validation\ValidatesRequests;
use Illuminate\Routing\Controller as BaseController;
class Controller extends BaseController
{
use AuthorizesRequests, ValidatesRequests;
}

68
scraper/app/Http/Kernel.php
View File

@ -1,68 +0,0 @@
<?php
namespace App\Http;
use Illuminate\Foundation\Http\Kernel as HttpKernel;
class Kernel extends HttpKernel
{
/**
* The application's global HTTP middleware stack.
*
* These middleware are run during every request to your application.
*
* @var array<int, class-string|string>
*/
protected $middleware = [
// \App\Http\Middleware\TrustHosts::class,
\App\Http\Middleware\TrustProxies::class,
\Illuminate\Http\Middleware\HandleCors::class,
\App\Http\Middleware\PreventRequestsDuringMaintenance::class,
\Illuminate\Foundation\Http\Middleware\ValidatePostSize::class,
\App\Http\Middleware\TrimStrings::class,
\Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull::class,
];
/**
* The application's route middleware groups.
*
* @var array<string, array<int, class-string|string>>
*/
protected $middlewareGroups = [
'web' => [
\App\Http\Middleware\EncryptCookies::class,
\Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse::class,
\Illuminate\Session\Middleware\StartSession::class,
\Illuminate\View\Middleware\ShareErrorsFromSession::class,
\App\Http\Middleware\VerifyCsrfToken::class,
\Illuminate\Routing\Middleware\SubstituteBindings::class,
],
'api' => [
// \Laravel\Sanctum\Http\Middleware\EnsureFrontendRequestsAreStateful::class,
\Illuminate\Routing\Middleware\ThrottleRequests::class.':api',
\Illuminate\Routing\Middleware\SubstituteBindings::class,
],
];
/**
* The application's middleware aliases.
*
* Aliases may be used instead of class names to conveniently assign middleware to routes and groups.
*
* @var array<string, class-string|string>
*/
protected $middlewareAliases = [
'auth' => \App\Http\Middleware\Authenticate::class,
'auth.basic' => \Illuminate\Auth\Middleware\AuthenticateWithBasicAuth::class,
'auth.session' => \Illuminate\Session\Middleware\AuthenticateSession::class,
'cache.headers' => \Illuminate\Http\Middleware\SetCacheHeaders::class,
'can' => \Illuminate\Auth\Middleware\Authorize::class,
'guest' => \App\Http\Middleware\RedirectIfAuthenticated::class,
'password.confirm' => \Illuminate\Auth\Middleware\RequirePassword::class,
'precognitive' => \Illuminate\Foundation\Http\Middleware\HandlePrecognitiveRequests::class,
'signed' => \App\Http\Middleware\ValidateSignature::class,
'throttle' => \Illuminate\Routing\Middleware\ThrottleRequests::class,
'verified' => \Illuminate\Auth\Middleware\EnsureEmailIsVerified::class,
];
}

17
scraper/app/Http/Middleware/Authenticate.php
View File

@ -1,17 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Auth\Middleware\Authenticate as Middleware;
use Illuminate\Http\Request;
class Authenticate extends Middleware
{
/**
* Get the path the user should be redirected to when they are not authenticated.
*/
protected function redirectTo(Request $request): ?string
{
return $request->expectsJson() ? null : route('login');
}
}

17
scraper/app/Http/Middleware/EncryptCookies.php
View File

@ -1,17 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Cookie\Middleware\EncryptCookies as Middleware;
class EncryptCookies extends Middleware
{
/**
* The names of the cookies that should not be encrypted.
*
* @var array<int, string>
*/
protected $except = [
//
];
}

17
scraper/app/Http/Middleware/PreventRequestsDuringMaintenance.php
View File

@ -1,17 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance as Middleware;
class PreventRequestsDuringMaintenance extends Middleware
{
/**
* The URIs that should be reachable while maintenance mode is enabled.
*
* @var array<int, string>
*/
protected $except = [
//
];
}

30
scraper/app/Http/Middleware/RedirectIfAuthenticated.php
View File

@ -1,30 +0,0 @@
<?php
namespace App\Http\Middleware;
use App\Providers\RouteServiceProvider;
use Closure;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Auth;
use Symfony\Component\HttpFoundation\Response;
class RedirectIfAuthenticated
{
/**
* Handle an incoming request.
*
* @param \Closure(\Illuminate\Http\Request): (\Symfony\Component\HttpFoundation\Response) $next
*/
public function handle(Request $request, Closure $next, string ...$guards): Response
{
$guards = empty($guards) ? [null] : $guards;
foreach ($guards as $guard) {
if (Auth::guard($guard)->check()) {
return redirect(RouteServiceProvider::HOME);
}
}
return $next($request);
}
}

19
scraper/app/Http/Middleware/TrimStrings.php
View File

@ -1,19 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Foundation\Http\Middleware\TrimStrings as Middleware;
class TrimStrings extends Middleware
{
/**
* The names of the attributes that should not be trimmed.
*
* @var array<int, string>
*/
protected $except = [
'current_password',
'password',
'password_confirmation',
];
}

20
scraper/app/Http/Middleware/TrustHosts.php
View File

@ -1,20 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Http\Middleware\TrustHosts as Middleware;
class TrustHosts extends Middleware
{
/**
* Get the host patterns that should be trusted.
*
* @return array<int, string|null>
*/
public function hosts(): array
{
return [
$this->allSubdomainsOfApplicationUrl(),
];
}
}

28
scraper/app/Http/Middleware/TrustProxies.php
View File

@ -1,28 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Http\Middleware\TrustProxies as Middleware;
use Illuminate\Http\Request;
class TrustProxies extends Middleware
{
/**
* The trusted proxies for this application.
*
* @var array<int, string>|string|null
*/
protected $proxies;
/**
* The headers that should be used to detect proxies.
*
* @var int
*/
protected $headers =
Request::HEADER_X_FORWARDED_FOR |
Request::HEADER_X_FORWARDED_HOST |
Request::HEADER_X_FORWARDED_PORT |
Request::HEADER_X_FORWARDED_PROTO |
Request::HEADER_X_FORWARDED_AWS_ELB;
}

22
scraper/app/Http/Middleware/ValidateSignature.php
View File

@ -1,22 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Routing\Middleware\ValidateSignature as Middleware;
class ValidateSignature extends Middleware
{
/**
* The names of the query string parameters that should be ignored.
*
* @var array<int, string>
*/
protected $except = [
// 'fbclid',
// 'utm_campaign',
// 'utm_content',
// 'utm_medium',
// 'utm_source',
// 'utm_term',
];
}

17
scraper/app/Http/Middleware/VerifyCsrfToken.php
View File

@ -1,17 +0,0 @@
<?php
namespace App\Http\Middleware;
use Illuminate\Foundation\Http\Middleware\VerifyCsrfToken as Middleware;
class VerifyCsrfToken extends Middleware
{
/**
* The URIs that should be excluded from CSRF verification.
*
* @var array<int, string>
*/
protected $except = [
//
];
}

33
scraper/app/Jobs/ScrapeProperty.php
View File

@ -1,33 +0,0 @@
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use App\Scraper\Edomizil;
class ScrapeProperty implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $uri;
/**
* Create a new job instance.
*/
public function __construct($uri)
{
$this->uri = $uri;
}
/**
* Execute the job.
*/
public function handle(): void
{
Edomizil::scrapeProperty($this->uri);
}
}

33
scraper/app/Jobs/ScrapePropertyData.php
View File

@ -1,33 +0,0 @@
<?php
namespace App\Jobs;
use App\Scraper\Edomizil;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class ScrapePropertyData implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $property;
/**
* Create a new job instance.
*/
public function __construct($property)
{
$this->property = $property;
}
/**
* Execute the job.
*/
public function handle(): void
{
Edomizil::scrapePropertyData($this->property);
sleep(random_int(1,10));
}
}

12
scraper/app/Models/Exception.php
View File

@ -1,12 +0,0 @@
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
class Exception extends Model
{
use HasFactory;
protected $table = 'exceptions';
protected $fillable = ['exception', 'type', 'entity_id'];
}

13
scraper/app/Models/Extraction.php
View File

@ -1,13 +0,0 @@
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
class Extraction extends Model
{
use HasFactory;
protected $table = 'extractions';
protected $fillable = ['property_id', 'type', 'body', 'header'];
}

13
scraper/app/Models/Property.php
View File

@ -1,13 +0,0 @@
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
class Property extends Model
{
use HasFactory;
protected $table = 'properties';
protected $fillable = ['property_platform_id', 'seed_id', 'check_data', 'last_found'];
}

13
scraper/app/Models/Regions.php
View File

@ -1,13 +0,0 @@
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
class Regions extends Model
{
use HasFactory;
protected $table = 'regions';
protected $fillable = ['name'];
}

19
scraper/app/Models/Seed.php
View File

@ -1,19 +0,0 @@
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Database\Eloquent\Model;
class Seed extends Model
{
use HasFactory;
protected $table = 'seeds';
protected $fillable = [
'region_id',
'uri'
];
}

47
scraper/app/Models/User.php
View File

@ -1,47 +0,0 @@
<?php
namespace App\Models;
// use Illuminate\Contracts\Auth\MustVerifyEmail;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Foundation\Auth\User as Authenticatable;
use Illuminate\Notifications\Notifiable;
class User extends Authenticatable
{
use HasFactory, Notifiable;
/**
* The attributes that are mass assignable.
*
* @var array<int, string>
*/
protected $fillable = [
'name',
'email',
'password',
];
/**
* The attributes that should be hidden for serialization.
*
* @var array<int, string>
*/
protected $hidden = [
'password',
'remember_token',
];
/**
* Get the attributes that should be cast.
*
* @return array<string, string>
*/
protected function casts(): array
{
return [
'email_verified_at' => 'datetime',
'password' => 'hashed',
];
}
}

24
scraper/app/Providers/AppServiceProvider.php
View File

@ -1,24 +0,0 @@
<?php
namespace App\Providers;
use Illuminate\Support\ServiceProvider;
class AppServiceProvider extends ServiceProvider
{
/**
* Register any application services.
*/
public function register(): void
{
//
}
/**
* Bootstrap any application services.
*/
public function boot(): void
{
//
}
}

26
scraper/app/Providers/AuthServiceProvider.php
View File

@ -1,26 +0,0 @@
<?php
namespace App\Providers;
// use Illuminate\Support\Facades\Gate;
use Illuminate\Foundation\Support\Providers\AuthServiceProvider as ServiceProvider;
class AuthServiceProvider extends ServiceProvider
{
/**
* The model to policy mappings for the application.
*
* @var array<class-string, class-string>
*/
protected $policies = [
//
];
/**
* Register any authentication / authorization services.
*/
public function boot(): void
{
//
}
}

19
scraper/app/Providers/BroadcastServiceProvider.php
View File

@ -1,19 +0,0 @@
<?php
namespace App\Providers;
use Illuminate\Support\Facades\Broadcast;
use Illuminate\Support\ServiceProvider;
class BroadcastServiceProvider extends ServiceProvider
{
/**
* Bootstrap any application services.
*/
public function boot(): void
{
Broadcast::routes();
require base_path('routes/channels.php');
}
}

38
scraper/app/Providers/EventServiceProvider.php
View File

@ -1,38 +0,0 @@
<?php
namespace App\Providers;
use Illuminate\Auth\Events\Registered;
use Illuminate\Auth\Listeners\SendEmailVerificationNotification;
use Illuminate\Foundation\Support\Providers\EventServiceProvider as ServiceProvider;
use Illuminate\Support\Facades\Event;
class EventServiceProvider extends ServiceProvider
{
/**
* The event to listener mappings for the application.
*
* @var array<class-string, array<int, class-string>>
*/
protected $listen = [
Registered::class => [
SendEmailVerificationNotification::class,
],
];
/**
* Register any events for your application.
*/
public function boot(): void
{
//
}
/**
* Determine if events and listeners should be automatically discovered.
*/
public function shouldDiscoverEvents(): bool
{
return false;
}
}

40
scraper/app/Providers/RouteServiceProvider.php
View File

@ -1,40 +0,0 @@
<?php
namespace App\Providers;
use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Foundation\Support\Providers\RouteServiceProvider as ServiceProvider;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\RateLimiter;
use Illuminate\Support\Facades\Route;
class RouteServiceProvider extends ServiceProvider
{
/**
* The path to your application's "home" route.
*
* Typically, users are redirected here after authentication.
*
* @var string
*/
public const HOME = '/home';
/**
* Define your route model bindings, pattern filters, and other route configuration.
*/
public function boot(): void
{
RateLimiter::for('api', function (Request $request) {
return Limit::perMinute(60)->by($request->user()?->id ?: $request->ip());
});
$this->routes(function () {
Route::middleware('api')
->prefix('api')
->group(base_path('routes/api.php'));
Route::middleware('web')
->group(base_path('routes/web.php'));
});
}
}

172
scraper/app/Scraper/Edomizil.php
View File

@ -1,172 +0,0 @@
<?php
namespace App\Scraper;
use App\Models\Seed;
use App\Models\Property;
use App\Models\Extraction;
use App\Models\Exception;
use App\Jobs\ScrapeProperty;
use App\Jobs\ScrapePropertyData;
use Illuminate\Support\Facades\Http;
class Edomizil{
public static function saveHttpException($response, $type, $entityId){
$exception = [];
$exception['status'] = $response->status();
$exception['headers'] = $response->headers();
$exception['body'] = $response->body();
$exceptionJSON = json_encode($exception);
Exception::create([
'exception' => $exceptionJSON,
'type' => $type,
'entity_id' => $entityId
]);
}
public static function getAllSeeds()
{
// get all seeds from model in random order.
return Seed::select('id','uri')->inRandomOrder()->get();
}
public static function getAllProperties()
{
// get all properties from model in random order.
return Property::select('id','property_platform_id')->inRandomOrder()->get();
}
public static function dispatchPropertyJobs()
{
$seeds = self::getAllSeeds();
foreach($seeds as $seed){
ScrapeProperty::dispatch($seed);
}
}
public static function dispatchPropertyDataJobs()
{
$properties = self::getAllProperties();
foreach($properties as $property){
ScrapePropertyData::dispatch($property);
}
}
public static function scrapeProperty($seed)
{
$response = Http::get($seed->uri);
if($response->successful()){
$json = $response->json();
foreach($json['offers'] as $offer){
$property = Property::firstWhere('property_platform_id', $offer['id']);
$geoLocation = implode(',', $offer['geoLocation']);
if($property){
$property->last_found = now();
$property->save();
// check if geoLocation is the same as last crawl
if($property->check_data !== $geoLocation){
Exception::create([
'exception' => 'geoLocation was different: '.$geoLocation,
'entity_type' => 'property',
'entity_id' => $property->id
]);
}
}else{
Property::create([
'property_platform_id' => $offer['id'],
'seed_id' => $seed->id,
'check_data' => $geoLocation,
'last_found' => now()
]);
}
}
return count($json['offers']);
}else{
self::saveHttpException($response,'property', $seed->id);
return 0;
}
}
public static function scrapePropertyData($property){
$result = [];
// scrape offer details such as name etc.
$offer = Http::get('https://www.e-domizil.ch/rental/offer/'.$property->property_platform_id);
if($offer->successful()){
Extraction::create([
'property_id' => $property->id,
'type' => 'offer',
'body' => $offer->body(),
'header' => json_encode($offer->headers())
]);
}else{
self::saveHttpException($offer,'offer',$property->id);
}
$result['offer'] = $offer->body();
// scrape price of property
$price = Http::withHeaders([
'Cookie' => 'c=CHF',
])->get('https://www.e-domizil.ch/booking/checkout/priceDetails/'.$property->property_platform_id);
if($price->successful()){
Extraction::create([
'property_id' => $property->id,
'type' => 'price',
'body' => $price->body(),
'header' => json_encode($price->headers())
]);
}else{
self::saveHttpException($price,'price',$property->id);
}
$result['price'] = $price->body();
// scrape calendar which contains occupancies
$calendar = Http::get('https://www.e-domizil.ch/api/v2/calendar/'.$property->property_platform_id, [
'year' => date("Y"),
'month' => date("m")
]);
if($calendar->successful()){
Extraction::create([
'property_id' => $property->id,
'type' => 'calendar',
'body' => $calendar->body(),
'header' => json_encode($calendar->headers())
]);
}else{
self::saveHttpException($calendar,'calendar',$property->id);
}
$result['calendar'] = $calendar->body();
return json_encode($result);
}
}

53
scraper/artisan
View File

@ -1,53 +0,0 @@
#!/usr/bin/env php
<?php
define('LARAVEL_START', microtime(true));
/*
|--------------------------------------------------------------------------
| Register The Auto Loader
|--------------------------------------------------------------------------
|
| Composer provides a convenient, automatically generated class loader
| for our application. We just need to utilize it! We'll require it
| into the script here so that we do not have to worry about the
| loading of any of our classes manually. It's great to relax.
|
*/
require __DIR__.'/vendor/autoload.php';
$app = require_once __DIR__.'/bootstrap/app.php';
/*
|--------------------------------------------------------------------------
| Run The Artisan Application
|--------------------------------------------------------------------------
|
| When we run the console application, the current CLI command will be
| executed in this console and the response sent back to a terminal
| or another output device for the developers. Here goes nothing!
|
*/
$kernel = $app->make(Illuminate\Contracts\Console\Kernel::class);
$status = $kernel->handle(
$input = new Symfony\Component\Console\Input\ArgvInput,
new Symfony\Component\Console\Output\ConsoleOutput
);
/*
|--------------------------------------------------------------------------
| Shutdown The Application
|--------------------------------------------------------------------------
|
| Once Artisan has finished running, we will fire off the shutdown events
| so that any final work may be done by the application before we shut
| down the process. This is the last thing to happen to the request.
|
*/
$kernel->terminate($input, $status);
exit($status);

55
scraper/bootstrap/app.php
View File

@ -1,55 +0,0 @@
<?php
/*
|--------------------------------------------------------------------------
| Create The Application
|--------------------------------------------------------------------------
|
| The first thing we will do is create a new Laravel application instance
| which serves as the "glue" for all the components of Laravel, and is
| the IoC container for the system binding all of the various parts.
|
*/
$app = new Illuminate\Foundation\Application(
$_ENV['APP_BASE_PATH'] ?? dirname(__DIR__)
);
/*
|--------------------------------------------------------------------------
| Bind Important Interfaces
|--------------------------------------------------------------------------
|
| Next, we need to bind some important interfaces into the container so
| we will be able to resolve them when needed. The kernels serve the
| incoming requests to this application from both the web and CLI.
|
*/
$app->singleton(
Illuminate\Contracts\Http\Kernel::class,
App\Http\Kernel::class
);
$app->singleton(
Illuminate\Contracts\Console\Kernel::class,
App\Console\Kernel::class
);
$app->singleton(
Illuminate\Contracts\Debug\ExceptionHandler::class,
App\Exceptions\Handler::class
);
/*
|--------------------------------------------------------------------------
| Return The Application
|--------------------------------------------------------------------------
|
| This script returns the application instance. The instance is given to
| the calling script so we can separate the building of the instances
| from the actual running of the application and sending responses.
|
*/
return $app;

2
scraper/bootstrap/cache/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

66
scraper/composer.json
View File

@ -1,66 +0,0 @@
{
"name": "laravel/laravel",
"type": "project",
"description": "The skeleton application for the Laravel framework.",
"keywords": ["laravel", "framework"],
"license": "MIT",
"require": {
"php": "^8.1",
"guzzlehttp/guzzle": "^7.2",
"laravel/framework": "^10.10",
"laravel/sanctum": "^3.3",
"laravel/tinker": "^2.8"
},
"require-dev": {
"fakerphp/faker": "^1.9.1",
"laravel/pint": "^1.0",
"laravel/sail": "^1.18",
"mockery/mockery": "^1.4.4",
"nunomaduro/collision": "^7.0",
"phpunit/phpunit": "^10.1",
"spatie/laravel-ignition": "^2.0"
},
"autoload": {
"psr-4": {
"App\\": "app/",
"Database\\Factories\\": "database/factories/",
"Database\\Seeders\\": "database/seeders/"
}
},
"autoload-dev": {
"psr-4": {
"Tests\\": "tests/"
}
},
"scripts": {
"post-autoload-dump": [
"Illuminate\\Foundation\\ComposerScripts::postAutoloadDump",
"@php artisan package:discover --ansi"
],
"post-update-cmd": [
"@php artisan vendor:publish --tag=laravel-assets --ansi --force"
],
"post-root-package-install": [
"@php -r \"file_exists('.env') || copy('.env.example', '.env');\""
],
"post-create-project-cmd": [
"@php artisan key:generate --ansi"
]
},
"extra": {
"laravel": {
"dont-discover": []
}
},
"config": {
"optimize-autoloader": true,
"preferred-install": "dist",
"sort-packages": true,
"allow-plugins": {
"pestphp/pest-plugin": true,
"php-http/discovery": true
}
},
"minimum-stability": "stable",
"prefer-stable": true
}

8202
scraper/composer.lock generated
View File

File diff suppressed because it is too large Load Diff

188
scraper/config/app.php
View File

@ -1,188 +0,0 @@
<?php
use Illuminate\Support\Facades\Facade;
use Illuminate\Support\ServiceProvider;
return [
/*
|--------------------------------------------------------------------------
| Application Name
|--------------------------------------------------------------------------
|
| This value is the name of your application. This value is used when the
| framework needs to place the application's name in a notification or
| any other location as required by the application or its packages.
|
*/
'name' => env('APP_NAME', 'Laravel'),
/*
|--------------------------------------------------------------------------
| Application Environment
|--------------------------------------------------------------------------
|
| This value determines the "environment" your application is currently
| running in. This may determine how you prefer to configure various
| services the application utilizes. Set this in your ".env" file.
|
*/
'env' => env('APP_ENV', 'production'),
/*
|--------------------------------------------------------------------------
| Application Debug Mode
|--------------------------------------------------------------------------
|
| When your application is in debug mode, detailed error messages with
| stack traces will be shown on every error that occurs within your
| application. If disabled, a simple generic error page is shown.
|
*/
'debug' => (bool) env('APP_DEBUG', false),
/*
|--------------------------------------------------------------------------
| Application URL
|--------------------------------------------------------------------------
|
| This URL is used by the console to properly generate URLs when using
| the Artisan command line tool. You should set this to the root of
| your application so that it is used when running Artisan tasks.
|
*/
'url' => env('APP_URL', 'http://localhost'),
'asset_url' => env('ASSET_URL'),
/*
|--------------------------------------------------------------------------
| Application Timezone
|--------------------------------------------------------------------------
|
| Here you may specify the default timezone for your application, which
| will be used by the PHP date and date-time functions. We have gone
| ahead and set this to a sensible default for you out of the box.
|
*/
'timezone' => 'UTC',
/*
|--------------------------------------------------------------------------
| Application Locale Configuration
|--------------------------------------------------------------------------
|
| The application locale determines the default locale that will be used
| by the translation service provider. You are free to set this value
| to any of the locales which will be supported by the application.
|
*/
'locale' => 'en',
/*
|--------------------------------------------------------------------------
| Application Fallback Locale
|--------------------------------------------------------------------------
|
| The fallback locale determines the locale to use when the current one
| is not available. You may change the value to correspond to any of
| the language folders that are provided through your application.
|
*/
'fallback_locale' => 'en',
/*
|--------------------------------------------------------------------------
| Faker Locale
|--------------------------------------------------------------------------
|
| This locale will be used by the Faker PHP library when generating fake
| data for your database seeds. For example, this will be used to get
| localized telephone numbers, street address information and more.
|
*/
'faker_locale' => 'en_US',
/*
|--------------------------------------------------------------------------
| Encryption Key
|--------------------------------------------------------------------------
|
| This key is used by the Illuminate encrypter service and should be set
| to a random, 32 character string, otherwise these encrypted strings
| will not be safe. Please do this before deploying an application!
|
*/
'key' => env('APP_KEY'),
'cipher' => 'AES-256-CBC',
/*
|--------------------------------------------------------------------------
| Maintenance Mode Driver
|--------------------------------------------------------------------------
|
| These configuration options determine the driver used to determine and
| manage Laravel's "maintenance mode" status. The "cache" driver will
| allow maintenance mode to be controlled across multiple machines.
|
| Supported drivers: "file", "cache"
|
*/
'maintenance' => [
'driver' => 'file',
// 'store' => 'redis',
],
/*
|--------------------------------------------------------------------------
| Autoloaded Service Providers
|--------------------------------------------------------------------------
|
| The service providers listed here will be automatically loaded on the
| request to your application. Feel free to add your own services to
| this array to grant expanded functionality to your applications.
|
*/
'providers' => ServiceProvider::defaultProviders()->merge([
/*
* Package Service Providers...
*/
/*
* Application Service Providers...
*/
App\Providers\AppServiceProvider::class,
App\Providers\AuthServiceProvider::class,
// App\Providers\BroadcastServiceProvider::class,
App\Providers\EventServiceProvider::class,
App\Providers\RouteServiceProvider::class,
])->toArray(),
/*
|--------------------------------------------------------------------------
| Class Aliases
|--------------------------------------------------------------------------
|
| This array of class aliases will be registered when this application
| is started. However, feel free to register as many as you wish as
| the aliases are "lazy" loaded so they don't hinder performance.
|
*/
'aliases' => Facade::defaultAliases()->merge([
// 'Example' => App\Facades\Example::class,
])->toArray(),
];

115
scraper/config/auth.php
View File

@ -1,115 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Authentication Defaults
|--------------------------------------------------------------------------
|
| This option controls the default authentication "guard" and password
| reset options for your application. You may change these defaults
| as required, but they're a perfect start for most applications.
|
*/
'defaults' => [
'guard' => 'web',
'passwords' => 'users',
],
/*
|--------------------------------------------------------------------------
| Authentication Guards
|--------------------------------------------------------------------------
|
| Next, you may define every authentication guard for your application.
| Of course, a great default configuration has been defined for you
| here which uses session storage and the Eloquent user provider.
|
| All authentication drivers have a user provider. This defines how the
| users are actually retrieved out of your database or other storage
| mechanisms used by this application to persist your user's data.
|
| Supported: "session"
|
*/
'guards' => [
'web' => [
'driver' => 'session',
'provider' => 'users',
],
],
/*
|--------------------------------------------------------------------------
| User Providers
|--------------------------------------------------------------------------
|
| All authentication drivers have a user provider. This defines how the
| users are actually retrieved out of your database or other storage
| mechanisms used by this application to persist your user's data.
|
| If you have multiple user tables or models you may configure multiple
| sources which represent each model / table. These sources may then
| be assigned to any extra authentication guards you have defined.
|
| Supported: "database", "eloquent"
|
*/
'providers' => [
'users' => [
'driver' => 'eloquent',
'model' => App\Models\User::class,
],
// 'users' => [
// 'driver' => 'database',
// 'table' => 'users',
// ],
],
/*
|--------------------------------------------------------------------------
| Resetting Passwords
|--------------------------------------------------------------------------
|
| You may specify multiple password reset configurations if you have more
| than one user table or model in the application and you want to have
| separate password reset settings based on the specific user types.
|
| The expiry time is the number of minutes that each reset token will be
| considered valid. This security feature keeps tokens short-lived so
| they have less time to be guessed. You may change this as needed.
|
| The throttle setting is the number of seconds a user must wait before
| generating more password reset tokens. This prevents the user from
| quickly generating a very large amount of password reset tokens.
|
*/
'passwords' => [
'users' => [
'provider' => 'users',
'table' => 'password_reset_tokens',
'expire' => 60,
'throttle' => 60,
],
],
/*
|--------------------------------------------------------------------------
| Password Confirmation Timeout
|--------------------------------------------------------------------------
|
| Here you may define the amount of seconds before a password confirmation
| times out and the user is prompted to re-enter their password via the
| confirmation screen. By default, the timeout lasts for three hours.
|
*/
'password_timeout' => 10800,
];

71
scraper/config/broadcasting.php
View File

@ -1,71 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Broadcaster
|--------------------------------------------------------------------------
|
| This option controls the default broadcaster that will be used by the
| framework when an event needs to be broadcast. You may set this to
| any of the connections defined in the "connections" array below.
|
| Supported: "pusher", "ably", "redis", "log", "null"
|
*/
'default' => env('BROADCAST_DRIVER', 'null'),
/*
|--------------------------------------------------------------------------
| Broadcast Connections
|--------------------------------------------------------------------------
|
| Here you may define all of the broadcast connections that will be used
| to broadcast events to other systems or over websockets. Samples of
| each available type of connection are provided inside this array.
|
*/
'connections' => [
'pusher' => [
'driver' => 'pusher',
'key' => env('PUSHER_APP_KEY'),
'secret' => env('PUSHER_APP_SECRET'),
'app_id' => env('PUSHER_APP_ID'),
'options' => [
'cluster' => env('PUSHER_APP_CLUSTER'),
'host' => env('PUSHER_HOST') ?: 'api-'.env('PUSHER_APP_CLUSTER', 'mt1').'.pusher.com',
'port' => env('PUSHER_PORT', 443),
'scheme' => env('PUSHER_SCHEME', 'https'),
'encrypted' => true,
'useTLS' => env('PUSHER_SCHEME', 'https') === 'https',
],
'client_options' => [
// Guzzle client options: https://docs.guzzlephp.org/en/stable/request-options.html
],
],
'ably' => [
'driver' => 'ably',
'key' => env('ABLY_KEY'),
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
],
'log' => [
'driver' => 'log',
],
'null' => [
'driver' => 'null',
],
],
];

111
scraper/config/cache.php
View File

@ -1,111 +0,0 @@
<?php
use Illuminate\Support\Str;
return [
/*
|--------------------------------------------------------------------------
| Default Cache Store
|--------------------------------------------------------------------------
|
| This option controls the default cache connection that gets used while
| using this caching library. This connection is used when another is
| not explicitly specified when executing a given caching function.
|
*/
'default' => env('CACHE_DRIVER', 'file'),
/*
|--------------------------------------------------------------------------
| Cache Stores
|--------------------------------------------------------------------------
|
| Here you may define all of the cache "stores" for your application as
| well as their drivers. You may even define multiple stores for the
| same cache driver to group types of items stored in your caches.
|
| Supported drivers: "apc", "array", "database", "file",
| "memcached", "redis", "dynamodb", "octane", "null"
|
*/
'stores' => [
'apc' => [
'driver' => 'apc',
],
'array' => [
'driver' => 'array',
'serialize' => false,
],
'database' => [
'driver' => 'database',
'table' => 'cache',
'connection' => null,
'lock_connection' => null,
],
'file' => [
'driver' => 'file',
'path' => storage_path('framework/cache/data'),
'lock_path' => storage_path('framework/cache/data'),
],
'memcached' => [
'driver' => 'memcached',
'persistent_id' => env('MEMCACHED_PERSISTENT_ID'),
'sasl' => [
env('MEMCACHED_USERNAME'),
env('MEMCACHED_PASSWORD'),
],
'options' => [
// Memcached::OPT_CONNECT_TIMEOUT => 2000,
],
'servers' => [
[
'host' => env('MEMCACHED_HOST', '127.0.0.1'),
'port' => env('MEMCACHED_PORT', 11211),
'weight' => 100,
],
],
],
'redis' => [
'driver' => 'redis',
'connection' => 'cache',
'lock_connection' => 'default',
],
'dynamodb' => [
'driver' => 'dynamodb',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'table' => env('DYNAMODB_CACHE_TABLE', 'cache'),
'endpoint' => env('DYNAMODB_ENDPOINT'),
],
'octane' => [
'driver' => 'octane',
],
],
/*
|--------------------------------------------------------------------------
| Cache Key Prefix
|--------------------------------------------------------------------------
|
| When utilizing the APC, database, memcached, Redis, or DynamoDB cache
| stores there might be other applications using the same cache. For
| that reason, you may prefix every cache key to avoid collisions.
|
*/
'prefix' => env('CACHE_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_cache_'),
];

34
scraper/config/cors.php
View File

@ -1,34 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Cross-Origin Resource Sharing (CORS) Configuration
|--------------------------------------------------------------------------
|
| Here you may configure your settings for cross-origin resource sharing
| or "CORS". This determines what cross-origin operations may execute
| in web browsers. You are free to adjust these settings as needed.
|
| To learn more: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
|
*/
'paths' => ['api/*', 'sanctum/csrf-cookie'],
'allowed_methods' => ['*'],
'allowed_origins' => ['*'],
'allowed_origins_patterns' => [],
'allowed_headers' => ['*'],
'exposed_headers' => [],
'max_age' => 0,
'supports_credentials' => false,
];

151
scraper/config/database.php
View File

@ -1,151 +0,0 @@
<?php
use Illuminate\Support\Str;
return [
/*
|--------------------------------------------------------------------------
| Default Database Connection Name
|--------------------------------------------------------------------------
|
| Here you may specify which of the database connections below you wish
| to use as your default connection for all database work. Of course
| you may use many connections at once using the Database library.
|
*/
'default' => env('DB_CONNECTION', 'mysql'),
/*
|--------------------------------------------------------------------------
| Database Connections
|--------------------------------------------------------------------------
|
| Here are each of the database connections setup for your application.
| Of course, examples of configuring each database platform that is
| supported by Laravel is shown below to make development simple.
|
|
| All database work in Laravel is done through the PHP PDO facilities
| so make sure you have the driver for your particular database of
| choice installed on your machine before you begin development.
|
*/
'connections' => [
'sqlite' => [
'driver' => 'sqlite',
'url' => env('DATABASE_URL'),
'database' => env('DB_DATABASE', database_path('database.sqlite')),
'prefix' => '',
'foreign_key_constraints' => env('DB_FOREIGN_KEYS', true),
],
'mysql' => [
'driver' => 'mysql',
'url' => env('DATABASE_URL'),
'host' => env('DB_HOST', '127.0.0.1'),
'port' => env('DB_PORT', '3306'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'unix_socket' => env('DB_SOCKET', ''),
'charset' => 'utf8mb4',
'collation' => 'utf8mb4_unicode_ci',
'prefix' => '',
'prefix_indexes' => true,
'strict' => true,
'engine' => null,
'options' => extension_loaded('pdo_mysql') ? array_filter([
PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'),
]) : [],
],
'pgsql' => [
'driver' => 'pgsql',
'url' => env('DATABASE_URL'),
'host' => env('DB_HOST', '127.0.0.1'),
'port' => env('DB_PORT', '5432'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'charset' => 'utf8',
'prefix' => '',
'prefix_indexes' => true,
'search_path' => 'public',
'sslmode' => 'prefer',
],
'sqlsrv' => [
'driver' => 'sqlsrv',
'url' => env('DATABASE_URL'),
'host' => env('DB_HOST', 'localhost'),
'port' => env('DB_PORT', '1433'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'charset' => 'utf8',
'prefix' => '',
'prefix_indexes' => true,
// 'encrypt' => env('DB_ENCRYPT', 'yes'),
// 'trust_server_certificate' => env('DB_TRUST_SERVER_CERTIFICATE', 'false'),
],
],
/*
|--------------------------------------------------------------------------
| Migration Repository Table
|--------------------------------------------------------------------------
|
| This table keeps track of all the migrations that have already run for
| your application. Using this information, we can determine which of
| the migrations on disk haven't actually been run in the database.
|
*/
'migrations' => 'migrations',
/*
|--------------------------------------------------------------------------
| Redis Databases
|--------------------------------------------------------------------------
|
| Redis is an open source, fast, and advanced key-value store that also
| provides a richer body of commands than a typical key-value system
| such as APC or Memcached. Laravel makes it easy to dig right in.
|
*/
'redis' => [
'client' => env('REDIS_CLIENT', 'phpredis'),
'options' => [
'cluster' => env('REDIS_CLUSTER', 'redis'),
'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
],
'default' => [
'url' => env('REDIS_URL'),
'host' => env('REDIS_HOST', '127.0.0.1'),
'username' => env('REDIS_USERNAME'),
'password' => env('REDIS_PASSWORD'),
'port' => env('REDIS_PORT', '6379'),
'database' => env('REDIS_DB', '0'),
],
'cache' => [
'url' => env('REDIS_URL'),
'host' => env('REDIS_HOST', '127.0.0.1'),
'username' => env('REDIS_USERNAME'),
'password' => env('REDIS_PASSWORD'),
'port' => env('REDIS_PORT', '6379'),
'database' => env('REDIS_CACHE_DB', '1'),
],
],
];

76
scraper/config/filesystems.php
View File

@ -1,76 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Filesystem Disk
|--------------------------------------------------------------------------
|
| Here you may specify the default filesystem disk that should be used
| by the framework. The "local" disk, as well as a variety of cloud
| based disks are available to your application. Just store away!
|
*/
'default' => env('FILESYSTEM_DISK', 'local'),
/*
|--------------------------------------------------------------------------
| Filesystem Disks
|--------------------------------------------------------------------------
|
| Here you may configure as many filesystem "disks" as you wish, and you
| may even configure multiple disks of the same driver. Defaults have
| been set up for each driver as an example of the required values.
|
| Supported Drivers: "local", "ftp", "sftp", "s3"
|
*/
'disks' => [
'local' => [
'driver' => 'local',
'root' => storage_path('app'),
'throw' => false,
],
'public' => [
'driver' => 'local',
'root' => storage_path('app/public'),
'url' => env('APP_URL').'/storage',
'visibility' => 'public',
'throw' => false,
],
's3' => [
'driver' => 's3',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'region' => env('AWS_DEFAULT_REGION'),
'bucket' => env('AWS_BUCKET'),
'url' => env('AWS_URL'),
'endpoint' => env('AWS_ENDPOINT'),
'use_path_style_endpoint' => env('AWS_USE_PATH_STYLE_ENDPOINT', false),
'throw' => false,
],
],
/*
|--------------------------------------------------------------------------
| Symbolic Links
|--------------------------------------------------------------------------
|
| Here you may configure the symbolic links that will be created when the
| `storage:link` Artisan command is executed. The array keys should be
| the locations of the links and the values should be their targets.
|
*/
'links' => [
public_path('storage') => storage_path('app/public'),
],
];

54
scraper/config/hashing.php
View File

@ -1,54 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Hash Driver
|--------------------------------------------------------------------------
|
| This option controls the default hash driver that will be used to hash
| passwords for your application. By default, the bcrypt algorithm is
| used; however, you remain free to modify this option if you wish.
|
| Supported: "bcrypt", "argon", "argon2id"
|
*/
'driver' => 'bcrypt',
/*
|--------------------------------------------------------------------------
| Bcrypt Options
|--------------------------------------------------------------------------
|
| Here you may specify the configuration options that should be used when
| passwords are hashed using the Bcrypt algorithm. This will allow you
| to control the amount of time it takes to hash the given password.
|
*/
'bcrypt' => [
'rounds' => env('BCRYPT_ROUNDS', 12),
'verify' => true,
],
/*
|--------------------------------------------------------------------------
| Argon Options
|--------------------------------------------------------------------------
|
| Here you may specify the configuration options that should be used when
| passwords are hashed using the Argon algorithm. These will allow you
| to control the amount of time it takes to hash the given password.
|
*/
'argon' => [
'memory' => 65536,
'threads' => 1,
'time' => 4,
'verify' => true,
],
];

131
scraper/config/logging.php
View File

@ -1,131 +0,0 @@
<?php
use Monolog\Handler\NullHandler;
use Monolog\Handler\StreamHandler;
use Monolog\Handler\SyslogUdpHandler;
use Monolog\Processor\PsrLogMessageProcessor;
return [
/*
|--------------------------------------------------------------------------
| Default Log Channel
|--------------------------------------------------------------------------
|
| This option defines the default log channel that gets used when writing
| messages to the logs. The name specified in this option should match
| one of the channels defined in the "channels" configuration array.
|
*/
'default' => env('LOG_CHANNEL', 'stack'),
/*
|--------------------------------------------------------------------------
| Deprecations Log Channel
|--------------------------------------------------------------------------
|
| This option controls the log channel that should be used to log warnings
| regarding deprecated PHP and library features. This allows you to get
| your application ready for upcoming major versions of dependencies.
|
*/
'deprecations' => [
'channel' => env('LOG_DEPRECATIONS_CHANNEL', 'null'),
'trace' => false,
],
/*
|--------------------------------------------------------------------------
| Log Channels
|--------------------------------------------------------------------------
|
| Here you may configure the log channels for your application. Out of
| the box, Laravel uses the Monolog PHP logging library. This gives
| you a variety of powerful log handlers / formatters to utilize.
|
| Available Drivers: "single", "daily", "slack", "syslog",
| "errorlog", "monolog",
| "custom", "stack"
|
*/
'channels' => [
'stack' => [
'driver' => 'stack',
'channels' => ['single'],
'ignore_exceptions' => false,
],
'single' => [
'driver' => 'single',
'path' => storage_path('logs/laravel.log'),
'level' => env('LOG_LEVEL', 'debug'),
'replace_placeholders' => true,
],
'daily' => [
'driver' => 'daily',
'path' => storage_path('logs/laravel.log'),
'level' => env('LOG_LEVEL', 'debug'),
'days' => 14,
'replace_placeholders' => true,
],
'slack' => [
'driver' => 'slack',
'url' => env('LOG_SLACK_WEBHOOK_URL'),
'username' => 'Laravel Log',
'emoji' => ':boom:',
'level' => env('LOG_LEVEL', 'critical'),
'replace_placeholders' => true,
],
'papertrail' => [
'driver' => 'monolog',
'level' => env('LOG_LEVEL', 'debug'),
'handler' => env('LOG_PAPERTRAIL_HANDLER', SyslogUdpHandler::class),
'handler_with' => [
'host' => env('PAPERTRAIL_URL'),
'port' => env('PAPERTRAIL_PORT'),
'connectionString' => 'tls://'.env('PAPERTRAIL_URL').':'.env('PAPERTRAIL_PORT'),
],
'processors' => [PsrLogMessageProcessor::class],
],
'stderr' => [
'driver' => 'monolog',
'level' => env('LOG_LEVEL', 'debug'),
'handler' => StreamHandler::class,
'formatter' => env('LOG_STDERR_FORMATTER'),
'with' => [
'stream' => 'php://stderr',
],
'processors' => [PsrLogMessageProcessor::class],
],
'syslog' => [
'driver' => 'syslog',
'level' => env('LOG_LEVEL', 'debug'),
'facility' => LOG_USER,
'replace_placeholders' => true,
],
'errorlog' => [
'driver' => 'errorlog',
'level' => env('LOG_LEVEL', 'debug'),
'replace_placeholders' => true,
],
'null' => [
'driver' => 'monolog',
'handler' => NullHandler::class,
],
'emergency' => [
'path' => storage_path('logs/laravel.log'),
],
],
];

134
scraper/config/mail.php
View File

@ -1,134 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Mailer
|--------------------------------------------------------------------------
|
| This option controls the default mailer that is used to send any email
| messages sent by your application. Alternative mailers may be setup
| and used as needed; however, this mailer will be used by default.
|
*/
'default' => env('MAIL_MAILER', 'smtp'),
/*
|--------------------------------------------------------------------------
| Mailer Configurations
|--------------------------------------------------------------------------
|
| Here you may configure all of the mailers used by your application plus
| their respective settings. Several examples have been configured for
| you and you are free to add your own as your application requires.
|
| Laravel supports a variety of mail "transport" drivers to be used while
| sending an e-mail. You will specify which one you are using for your
| mailers below. You are free to add additional mailers as required.
|
| Supported: "smtp", "sendmail", "mailgun", "ses", "ses-v2",
| "postmark", "log", "array", "failover", "roundrobin"
|
*/
'mailers' => [
'smtp' => [
'transport' => 'smtp',
'url' => env('MAIL_URL'),
'host' => env('MAIL_HOST', 'smtp.mailgun.org'),
'port' => env('MAIL_PORT', 587),
'encryption' => env('MAIL_ENCRYPTION', 'tls'),
'username' => env('MAIL_USERNAME'),
'password' => env('MAIL_PASSWORD'),
'timeout' => null,
'local_domain' => env('MAIL_EHLO_DOMAIN'),
],
'ses' => [
'transport' => 'ses',
],
'postmark' => [
'transport' => 'postmark',
// 'message_stream_id' => null,
// 'client' => [
// 'timeout' => 5,
// ],
],
'mailgun' => [
'transport' => 'mailgun',
// 'client' => [
// 'timeout' => 5,
// ],
],
'sendmail' => [
'transport' => 'sendmail',
'path' => env('MAIL_SENDMAIL_PATH', '/usr/sbin/sendmail -bs -i'),
],
'log' => [
'transport' => 'log',
'channel' => env('MAIL_LOG_CHANNEL'),
],
'array' => [
'transport' => 'array',
],
'failover' => [
'transport' => 'failover',
'mailers' => [
'smtp',
'log',
],
],
'roundrobin' => [
'transport' => 'roundrobin',
'mailers' => [
'ses',
'postmark',
],
],
],
/*
|--------------------------------------------------------------------------
| Global "From" Address
|--------------------------------------------------------------------------
|
| You may wish for all e-mails sent by your application to be sent from
| the same address. Here, you may specify a name and address that is
| used globally for all e-mails that are sent by your application.
|
*/
'from' => [
'address' => env('MAIL_FROM_ADDRESS', 'hello@example.com'),
'name' => env('MAIL_FROM_NAME', 'Example'),
],
/*
|--------------------------------------------------------------------------
| Markdown Mail Settings
|--------------------------------------------------------------------------
|
| If you are using Markdown based email rendering, you may configure your
| theme and component paths here, allowing you to customize the design
| of the emails. Or, you may simply stick with the Laravel defaults!
|
*/
'markdown' => [
'theme' => 'default',
'paths' => [
resource_path('views/vendor/mail'),
],
],
];

109
scraper/config/queue.php
View File

@ -1,109 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Queue Connection Name
|--------------------------------------------------------------------------
|
| Laravel's queue API supports an assortment of back-ends via a single
| API, giving you convenient access to each back-end using the same
| syntax for every one. Here you may define a default connection.
|
*/
'default' => env('QUEUE_CONNECTION', 'sync'),
/*
|--------------------------------------------------------------------------
| Queue Connections
|--------------------------------------------------------------------------
|
| Here you may configure the connection information for each server that
| is used by your application. A default configuration has been added
| for each back-end shipped with Laravel. You are free to add more.
|
| Drivers: "sync", "database", "beanstalkd", "sqs", "redis", "null"
|
*/
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'after_commit' => false,
],
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'retry_after' => 90,
'block_for' => 0,
'after_commit' => false,
],
'sqs' => [
'driver' => 'sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
'queue' => env('SQS_QUEUE', 'default'),
'suffix' => env('SQS_SUFFIX'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'after_commit' => false,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
'after_commit' => false,
],
],
/*
|--------------------------------------------------------------------------
| Job Batching
|--------------------------------------------------------------------------
|
| The following options configure the database and table that store job
| batching information. These options can be updated to any database
| connection and table which has been defined by your application.
|
*/
'batching' => [
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'job_batches',
],
/*
|--------------------------------------------------------------------------
| Failed Queue Jobs
|--------------------------------------------------------------------------
|
| These options configure the behavior of failed queue job logging so you
| can control which database and table are used to store the jobs that
| have failed. You may change them to any database / table you wish.
|
*/
'failed' => [
'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];

83
scraper/config/sanctum.php
View File

@ -1,83 +0,0 @@
<?php
use Laravel\Sanctum\Sanctum;
return [
/*
|--------------------------------------------------------------------------
| Stateful Domains
|--------------------------------------------------------------------------
|
| Requests from the following domains / hosts will receive stateful API
| authentication cookies. Typically, these should include your local
| and production domains which access your API via a frontend SPA.
|
*/
'stateful' => explode(',', env('SANCTUM_STATEFUL_DOMAINS', sprintf(
'%s%s',
'localhost,localhost:3000,127.0.0.1,127.0.0.1:8000,::1',
Sanctum::currentApplicationUrlWithPort()
))),
/*
|--------------------------------------------------------------------------
| Sanctum Guards
|--------------------------------------------------------------------------
|
| This array contains the authentication guards that will be checked when
| Sanctum is trying to authenticate a request. If none of these guards
| are able to authenticate the request, Sanctum will use the bearer
| token that's present on an incoming request for authentication.
|
*/
'guard' => ['web'],
/*
|--------------------------------------------------------------------------
| Expiration Minutes
|--------------------------------------------------------------------------
|
| This value controls the number of minutes until an issued token will be
| considered expired. This will override any values set in the token's
| "expires_at" attribute, but first-party sessions are not affected.
|
*/
'expiration' => null,
/*
|--------------------------------------------------------------------------
| Token Prefix
|--------------------------------------------------------------------------
|
| Sanctum can prefix new tokens in order to take advantage of numerous
| security scanning initiatives maintained by open source platforms
| that notify developers if they commit tokens into repositories.
|
| See: https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning
|
*/
'token_prefix' => env('SANCTUM_TOKEN_PREFIX', ''),
/*
|--------------------------------------------------------------------------
| Sanctum Middleware
|--------------------------------------------------------------------------
|
| When authenticating your first-party SPA with Sanctum you may need to
| customize some of the middleware Sanctum uses while processing the
| request. You may change the middleware listed below as required.
|
*/
'middleware' => [
'authenticate_session' => Laravel\Sanctum\Http\Middleware\AuthenticateSession::class,
'encrypt_cookies' => App\Http\Middleware\EncryptCookies::class,
'verify_csrf_token' => App\Http\Middleware\VerifyCsrfToken::class,
],
];

34
scraper/config/services.php
View File

@ -1,34 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Third Party Services
|--------------------------------------------------------------------------
|
| This file is for storing the credentials for third party services such
| as Mailgun, Postmark, AWS and more. This file provides the de facto
| location for this type of information, allowing packages to have
| a conventional file to locate the various service credentials.
|
*/
'mailgun' => [
'domain' => env('MAILGUN_DOMAIN'),
'secret' => env('MAILGUN_SECRET'),
'endpoint' => env('MAILGUN_ENDPOINT', 'api.mailgun.net'),
'scheme' => 'https',
],
'postmark' => [
'token' => env('POSTMARK_TOKEN'),
],
'ses' => [
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
],
];

214
scraper/config/session.php
View File

@ -1,214 +0,0 @@
<?php
use Illuminate\Support\Str;
return [
/*
|--------------------------------------------------------------------------
| Default Session Driver
|--------------------------------------------------------------------------
|
| This option controls the default session "driver" that will be used on
| requests. By default, we will use the lightweight native driver but
| you may specify any of the other wonderful drivers provided here.
|
| Supported: "file", "cookie", "database", "apc",
| "memcached", "redis", "dynamodb", "array"
|
*/
'driver' => env('SESSION_DRIVER', 'file'),
/*
|--------------------------------------------------------------------------
| Session Lifetime
|--------------------------------------------------------------------------
|
| Here you may specify the number of minutes that you wish the session
| to be allowed to remain idle before it expires. If you want them
| to immediately expire on the browser closing, set that option.
|
*/
'lifetime' => env('SESSION_LIFETIME', 120),
'expire_on_close' => false,
/*
|--------------------------------------------------------------------------
| Session Encryption
|--------------------------------------------------------------------------
|
| This option allows you to easily specify that all of your session data
| should be encrypted before it is stored. All encryption will be run
| automatically by Laravel and you can use the Session like normal.
|
*/
'encrypt' => false,
/*
|--------------------------------------------------------------------------
| Session File Location
|--------------------------------------------------------------------------
|
| When using the native session driver, we need a location where session
| files may be stored. A default has been set for you but a different
| location may be specified. This is only needed for file sessions.
|
*/
'files' => storage_path('framework/sessions'),
/*
|--------------------------------------------------------------------------
| Session Database Connection
|--------------------------------------------------------------------------
|
| When using the "database" or "redis" session drivers, you may specify a
| connection that should be used to manage these sessions. This should
| correspond to a connection in your database configuration options.
|
*/
'connection' => env('SESSION_CONNECTION'),
/*
|--------------------------------------------------------------------------
| Session Database Table
|--------------------------------------------------------------------------
|
| When using the "database" session driver, you may specify the table we
| should use to manage the sessions. Of course, a sensible default is
| provided for you; however, you are free to change this as needed.
|
*/
'table' => 'sessions',
/*
|--------------------------------------------------------------------------
| Session Cache Store
|--------------------------------------------------------------------------
|
| While using one of the framework's cache driven session backends you may
| list a cache store that should be used for these sessions. This value
| must match with one of the application's configured cache "stores".
|
| Affects: "apc", "dynamodb", "memcached", "redis"
|
*/
'store' => env('SESSION_STORE'),
/*
|--------------------------------------------------------------------------
| Session Sweeping Lottery
|--------------------------------------------------------------------------
|
| Some session drivers must manually sweep their storage location to get
| rid of old sessions from storage. Here are the chances that it will
| happen on a given request. By default, the odds are 2 out of 100.
|
*/
'lottery' => [2, 100],
/*
|--------------------------------------------------------------------------
| Session Cookie Name
|--------------------------------------------------------------------------
|
| Here you may change the name of the cookie used to identify a session
| instance by ID. The name specified here will get used every time a
| new session cookie is created by the framework for every driver.
|
*/
'cookie' => env(
'SESSION_COOKIE',
Str::slug(env('APP_NAME', 'laravel'), '_').'_session'
),
/*
|--------------------------------------------------------------------------
| Session Cookie Path
|--------------------------------------------------------------------------
|
| The session cookie path determines the path for which the cookie will
| be regarded as available. Typically, this will be the root path of
| your application but you are free to change this when necessary.
|
*/
'path' => '/',
/*
|--------------------------------------------------------------------------
| Session Cookie Domain
|--------------------------------------------------------------------------
|
| Here you may change the domain of the cookie used to identify a session
| in your application. This will determine which domains the cookie is
| available to in your application. A sensible default has been set.
|
*/
'domain' => env('SESSION_DOMAIN'),
/*
|--------------------------------------------------------------------------
| HTTPS Only Cookies
|--------------------------------------------------------------------------
|
| By setting this option to true, session cookies will only be sent back
| to the server if the browser has a HTTPS connection. This will keep
| the cookie from being sent to you when it can't be done securely.
|
*/
'secure' => env('SESSION_SECURE_COOKIE'),
/*
|--------------------------------------------------------------------------
| HTTP Access Only
|--------------------------------------------------------------------------
|
| Setting this value to true will prevent JavaScript from accessing the
| value of the cookie and the cookie will only be accessible through
| the HTTP protocol. You are free to modify this option if needed.
|
*/
'http_only' => true,
/*
|--------------------------------------------------------------------------
| Same-Site Cookies
|--------------------------------------------------------------------------
|
| This option determines how your cookies behave when cross-site requests
| take place, and can be used to mitigate CSRF attacks. By default, we
| will set this value to "lax" since this is a secure default value.
|
| Supported: "lax", "strict", "none", null
|
*/
'same_site' => 'lax',
/*
|--------------------------------------------------------------------------
| Partitioned Cookies
|--------------------------------------------------------------------------
|
| Setting this value to true will tie the cookie to the top-level site for
| a cross-site context. Partitioned cookies are accepted by the browser
| when flagged "secure" and the Same-Site attribute is set to "none".
|
*/
'partitioned' => false,
];

36
scraper/config/view.php
View File

@ -1,36 +0,0 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| View Storage Paths
|--------------------------------------------------------------------------
|
| Most templating systems load templates from disk. Here you may specify
| an array of paths that should be checked for your views. Of course
| the usual Laravel view path has already been registered for you.
|
*/
'paths' => [
resource_path('views'),
],
/*
|--------------------------------------------------------------------------
| Compiled View Path
|--------------------------------------------------------------------------
|
| This option determines where all the compiled Blade templates will be
| stored for your application. Typically, this is within the storage
| directory. However, as usual, you are free to change this value.
|
*/
'compiled' => env(
'VIEW_COMPILED_PATH',
realpath(storage_path('framework/views'))
),
];

1
scraper/database/.gitignore vendored
View File

@ -1 +0,0 @@
*.sqlite*

44
scraper/database/factories/UserFactory.php
View File

@ -1,44 +0,0 @@
<?php
namespace Database\Factories;
use Illuminate\Database\Eloquent\Factories\Factory;
use Illuminate\Support\Facades\Hash;
use Illuminate\Support\Str;
/**
* @extends \Illuminate\Database\Eloquent\Factories\Factory<\App\Models\User>
*/
class UserFactory extends Factory
{
/**
* The current password being used by the factory.
*/
protected static ?string $password;
/**
* Define the model's default state.
*
* @return array<string, mixed>
*/
public function definition(): array
{
return [
'name' => fake()->name(),
'email' => fake()->unique()->safeEmail(),
'email_verified_at' => now(),
'password' => static::$password ??= Hash::make('password'),
'remember_token' => Str::random(10),
];
}
/**
* Indicate that the model's email address should be unverified.
*/
public function unverified(): static
{
return $this->state(fn (array $attributes) => [
'email_verified_at' => null,
]);
}
}

49
scraper/database/migrations/0001_01_01_000000_create_users_table.php
View File

@ -1,49 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('users', function (Blueprint $table) {
$table->id();
$table->string('name');
$table->string('email')->unique();
$table->timestamp('email_verified_at')->nullable();
$table->string('password');
$table->rememberToken();
$table->timestamps();
});
Schema::create('password_reset_tokens', function (Blueprint $table) {
$table->string('email')->primary();
$table->string('token');
$table->timestamp('created_at')->nullable();
});
Schema::create('sessions', function (Blueprint $table) {
$table->string('id')->primary();
$table->foreignId('user_id')->nullable()->index();
$table->string('ip_address', 45)->nullable();
$table->text('user_agent')->nullable();
$table->longText('payload');
$table->integer('last_activity')->index();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('users');
Schema::dropIfExists('password_reset_tokens');
Schema::dropIfExists('sessions');
}
};

35
scraper/database/migrations/0001_01_01_000001_create_cache_table.php
View File

@ -1,35 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('cache', function (Blueprint $table) {
$table->string('key')->primary();
$table->mediumText('value');
$table->integer('expiration');
});
Schema::create('cache_locks', function (Blueprint $table) {
$table->string('key')->primary();
$table->string('owner');
$table->integer('expiration');
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('cache');
Schema::dropIfExists('cache_locks');
}
};

33
scraper/database/migrations/2019_12_14_000001_create_personal_access_tokens_table.php
View File

@ -1,33 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('personal_access_tokens', function (Blueprint $table) {
$table->id();
$table->morphs('tokenable');
$table->string('name');
$table->string('token', 64)->unique();
$table->text('abilities')->nullable();
$table->timestamp('last_used_at')->nullable();
$table->timestamp('expires_at')->nullable();
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('personal_access_tokens');
}
};

28
scraper/database/migrations/2024_03_15_142227_create_regions_table.php
View File

@ -1,28 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('regions', function (Blueprint $table) {
$table->id();
$table->string('name', 255);
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('regions');
}
};

29
scraper/database/migrations/2024_03_15_142228_create_seeds_table.php
View File

@ -1,29 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('seeds', function (Blueprint $table) {
$table->id();
$table->text('uri');
$table->foreignId('region_id')->constrained();
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('seeds');
}
};

31
scraper/database/migrations/2024_03_15_142257_create_properties_table.php
View File

@ -1,31 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('properties', function (Blueprint $table) {
$table->id();
$table->string('property_platform_id', 255);
$table->foreignId('seed_id')->constrained();
$table->text('check_data');
$table->timestamp('last_found', precision: 0);
$table->timestamps();
});
}
/**->references('id')->on('users');
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('properties');
}
};

31
scraper/database/migrations/2024_03_15_142550_create_extractions_table.php
View File

@ -1,31 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('extractions', function (Blueprint $table) {
$table->id();
$table->foreignId('property_id')->constrained();
$table->longtext('body');
$table->longtext('header');
$table->enum('type', ['property', 'calendar', 'offer', 'price']);
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('extractions');
}
};

30
scraper/database/migrations/2024_03_15_142625_create_exceptions_table.php
View File

@ -1,30 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('exceptions', function (Blueprint $table) {
$table->id();
$table->longtext('exception');
$table->enum('type', ['property', 'calendar', 'offer', 'price']);
$table->integer('entity_id');
$table->timestamps();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('exceptions');
}
};

32
scraper/database/migrations/2024_03_15_162023_create_jobs_table.php
View File

@ -1,32 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('jobs', function (Blueprint $table) {
$table->bigIncrements('id');
$table->string('queue')->index();
$table->longText('payload');
$table->unsignedTinyInteger('attempts');
$table->unsignedInteger('reserved_at')->nullable();
$table->unsignedInteger('available_at');
$table->unsignedInteger('created_at');
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('jobs');
}
};

32
scraper/database/migrations/2024_04_08_115153_create_failed_jobs_table.php
View File

@ -1,32 +0,0 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::create('failed_jobs', function (Blueprint $table) {
$table->id();
$table->string('uuid')->unique();
$table->text('connection');
$table->text('queue');
$table->longText('payload');
$table->longText('exception');
$table->timestamp('failed_at')->useCurrent();
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::dropIfExists('failed_jobs');
}
};

35
scraper/database/seeders/DatabaseSeeder.php
View File

@ -1,35 +0,0 @@
<?php
namespace Database\Seeders;
use App\Models\Regions;
use App\Models\Seeds;
use Illuminate\Database\Seeder;
use Illuminate\Support\Facades\DB;
class DatabaseSeeder extends Seeder
{
/**
* Seed the application's database.
*/
public function run(): void
{
// User::factory(10)->create();
DB::table('regions')->insert([
'name' => 'Heidiland',
]);
DB::table('seeds')->insert([
'uri' => 'https://www.e-domizil.ch/search/632d3fb65adbe?_format=json&adults=1&duration=7',
'region_id' => 1
]);
DB::table('properties')->insert([
'property_platform_id' => '12345',
'seed_id' => 1,
'check_data' => '12345',
'last_found' => '2024-01-01'
]);
}
}

13
scraper/package.json
View File

@ -1,13 +0,0 @@
{
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build"
},
"devDependencies": {
"axios": "^1.6.4",
"laravel-vite-plugin": "^1.0.0",
"vite": "^5.0.0"
}
}

32
scraper/phpunit.xml
View File

@ -1,32 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
bootstrap="vendor/autoload.php"
colors="true"
>
<testsuites>
<testsuite name="Unit">
<directory>tests/Unit</directory>
</testsuite>
<testsuite name="Feature">
<directory>tests/Feature</directory>
</testsuite>
</testsuites>
<source>
<include>
<directory>app</directory>
</include>
</source>
<php>
<env name="APP_ENV" value="testing"/>
<env name="BCRYPT_ROUNDS" value="4"/>
<env name="CACHE_DRIVER" value="array"/>
<!-- <env name="DB_CONNECTION" value="sqlite"/> -->
<!-- <env name="DB_DATABASE" value=":memory:"/> -->
<env name="MAIL_MAILER" value="array"/>
<env name="PULSE_ENABLED" value="false"/>
<env name="QUEUE_CONNECTION" value="sync"/>
<env name="SESSION_DRIVER" value="array"/>
<env name="TELESCOPE_ENABLED" value="false"/>
</php>
</phpunit>

21
scraper/public/.htaccess
View File

@ -1,21 +0,0 @@
<IfModule mod_rewrite.c>
<IfModule mod_negotiation.c>
Options -MultiViews -Indexes
</IfModule>
RewriteEngine On
# Handle Authorization Header
RewriteCond %{HTTP:Authorization} .
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
# Redirect Trailing Slashes If Not A Folder...
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} (.+)/$
RewriteRule ^ %1 [L,R=301]
# Send Requests To Front Controller...
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [L]
</IfModule>

0
scraper/public/favicon.ico
View File

55
scraper/public/index.php
View File

@ -1,55 +0,0 @@
<?php
use Illuminate\Contracts\Http\Kernel;
use Illuminate\Http\Request;
define('LARAVEL_START', microtime(true));
/*
|--------------------------------------------------------------------------
| Check If The Application Is Under Maintenance
|--------------------------------------------------------------------------
|
| If the application is in maintenance / demo mode via the "down" command
| we will load this file so that any pre-rendered content can be shown
| instead of starting the framework, which could cause an exception.
|
*/
if (file_exists($maintenance = __DIR__.'/../storage/framework/maintenance.php')) {
require $maintenance;
}
/*
|--------------------------------------------------------------------------
| Register The Auto Loader
|--------------------------------------------------------------------------
|
| Composer provides a convenient, automatically generated class loader for
| this application. We just need to utilize it! We'll simply require it
| into the script here so we don't need to manually load our classes.
|
*/
require __DIR__.'/../vendor/autoload.php';
/*
|--------------------------------------------------------------------------
| Run The Application
|--------------------------------------------------------------------------
|
| Once we have the application, we can handle the incoming request using
| the application's HTTP kernel. Then, we will send the response back
| to this client's browser, allowing them to enjoy our application.
|
*/
$app = require_once __DIR__.'/../bootstrap/app.php';
$kernel = $app->make(Kernel::class);
$response = $kernel->handle(
$request = Request::capture()
)->send();
$kernel->terminate($request, $response);

2
scraper/public/robots.txt
View File

@ -1,2 +0,0 @@
User-agent: *
Disallow:

0
scraper/resources/css/app.css
View File

1
scraper/resources/js/app.js
View File

@ -1 +0,0 @@
import './bootstrap';

32
scraper/resources/js/bootstrap.js
View File

@ -1,32 +0,0 @@
/**
* We'll load the axios HTTP library which allows us to easily issue requests
* to our Laravel back-end. This library automatically handles sending the
* CSRF token as a header based on the value of the "XSRF" token cookie.
*/
import axios from 'axios';
window.axios = axios;
window.axios.defaults.headers.common['X-Requested-With'] = 'XMLHttpRequest';
/**
* Echo exposes an expressive API for subscribing to channels and listening
* for events that are broadcast by Laravel. Echo and event broadcasting
* allows your team to easily build robust real-time web applications.
*/
// import Echo from 'laravel-echo';
// import Pusher from 'pusher-js';
// window.Pusher = Pusher;
// window.Echo = new Echo({
// broadcaster: 'pusher',
// key: import.meta.env.VITE_PUSHER_APP_KEY,
// cluster: import.meta.env.VITE_PUSHER_APP_CLUSTER ?? 'mt1',
// wsHost: import.meta.env.VITE_PUSHER_HOST ? import.meta.env.VITE_PUSHER_HOST : `ws-${import.meta.env.VITE_PUSHER_APP_CLUSTER}.pusher.com`,
// wsPort: import.meta.env.VITE_PUSHER_PORT ?? 80,
// wssPort: import.meta.env.VITE_PUSHER_PORT ?? 443,
// forceTLS: (import.meta.env.VITE_PUSHER_SCHEME ?? 'https') === 'https',
// enabledTransports: ['ws', 'wss'],
// });

133
scraper/resources/views/welcome.blade.php
View File

File diff suppressed because one or more lines are too long

19
scraper/routes/api.php
View File

@ -1,19 +0,0 @@
<?php
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Route;
/*
|--------------------------------------------------------------------------
| API Routes
|--------------------------------------------------------------------------
|
| Here is where you can register API routes for your application. These
| routes are loaded by the RouteServiceProvider and all of them will
| be assigned to the "api" middleware group. Make something great!
|
*/
Route::middleware('auth:sanctum')->get('/user', function (Request $request) {
return $request->user();
});

18
scraper/routes/channels.php
View File

@ -1,18 +0,0 @@
<?php
use Illuminate\Support\Facades\Broadcast;
/*
|--------------------------------------------------------------------------
| Broadcast Channels
|--------------------------------------------------------------------------
|
| Here you may register all of the event broadcasting channels that your
| application supports. The given channel authorization callbacks are
| used to check if an authenticated user can listen to the channel.
|
*/
Broadcast::channel('App.Models.User.{id}', function ($user, $id) {
return (int) $user->id === (int) $id;
});

19
scraper/routes/console.php
View File

@ -1,19 +0,0 @@
<?php
use Illuminate\Foundation\Inspiring;
use Illuminate\Support\Facades\Artisan;
/*
|--------------------------------------------------------------------------
| Console Routes
|--------------------------------------------------------------------------
|
| This file is where you may define all of your Closure based console
| commands. Each Closure is bound to a command instance allowing a
| simple approach to interacting with each command's IO methods.
|
*/
Artisan::command('inspire', function () {
$this->comment(Inspiring::quote());
})->purpose('Display an inspiring quote');

18
scraper/routes/web.php
View File

@ -1,18 +0,0 @@
<?php
use Illuminate\Support\Facades\Route;
/*
|--------------------------------------------------------------------------
| Web Routes
|--------------------------------------------------------------------------
|
| Here is where you can register web routes for your application. These
| routes are loaded by the RouteServiceProvider and all of them will
| be assigned to the "web" middleware group. Make something great!
|
*/
Route::get('/', function () {
return view('welcome');
});

3
scraper/storage/app/.gitignore vendored
View File

@ -1,3 +0,0 @@
*
!public/
!.gitignore

2
scraper/storage/app/public/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

9
scraper/storage/framework/.gitignore vendored
View File

@ -1,9 +0,0 @@
compiled.php
config.php
down
events.scanned.php
maintenance.php
routes.php
routes.scanned.php
schedule-*
services.json

3
scraper/storage/framework/cache/.gitignore vendored
View File

@ -1,3 +0,0 @@
*
!data/
!.gitignore

2
scraper/storage/framework/cache/data/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

2
scraper/storage/framework/sessions/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

2
scraper/storage/framework/testing/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

2
scraper/storage/framework/views/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

2
scraper/storage/logs/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

21
scraper/tests/CreatesApplication.php
View File

@ -1,21 +0,0 @@
<?php
namespace Tests;
use Illuminate\Contracts\Console\Kernel;
use Illuminate\Foundation\Application;
trait CreatesApplication
{
/**
* Creates the application.
*/
public function createApplication(): Application
{
$app = require __DIR__.'/../bootstrap/app.php';
$app->make(Kernel::class)->bootstrap();
return $app;
}
}

Some files were not shown because too many files have changed in this diff Show More