Initial support for new Spotify export format, GH-215

Refactor container image to use linuxserverio alpine base

.dockerignore

@@ -1,7 +1,7 @@
 *
 !maloja
+!container
 !Containerfile
-!requirements_pre.txt
 !requirements.txt
 !pyproject.toml
 !README.md

Containerfile

@@ -1,40 +1,74 @@
-FROM alpine:3.15
-# Python image includes two Python versions, so use base Alpine
-
-# Based on the work of Jonathan Boeckel <jonathanboeckel1996@gmail.com>
+FROM lsiobase/alpine:3.17 as base
 
 WORKDIR /usr/src/app
 
-# Install run dependencies first
-RUN apk add --no-cache python3 py3-lxml tzdata
 
-# system pip could be removed after build, but apk then decides to also remove all its
-# python dependencies, even if they are explicitly installed as python packages
-# whut
+
+COPY --chown=abc:abc ./requirements.txt ./requirements.txt
+
+# based on https://github.com/linuxserver/docker-pyload-ng/blob/main/Dockerfile
+# everything but the app installation is run in one command so we can purge
+# all build dependencies and cache in the same layer
+# it may be possible to decrease image size slightly by using build stage and
+# copying all site-packages to runtime stage but the image is already pretty small
 RUN \
-	apk add py3-pip && \
-	pip install wheel
+  echo "**** install build packages ****" && \
+	apk add --no-cache --virtual=build-deps \
+		gcc \
+		g++ \
+		python3-dev \
+		libxml2-dev \
+		libxslt-dev \
+		libffi-dev \
+		libc-dev \
+		py3-pip \
+		linux-headers && \
+	echo "**** install runtime packages ****" && \
+	apk add --no-cache \
+		python3 \
+		py3-lxml \
+		tzdata && \
+	echo "**** install pip dependencies ****" && \
+	python3 -m ensurepip && \
+  pip3 install -U --no-cache-dir \
+	  pip \
+	  wheel && \
+  echo "**** install maloja requirements ****" && \
+  pip3 install --no-cache-dir -r requirements.txt && \
+	echo "**** cleanup ****" && \
+	apk del --purge \
+		build-deps && \
+	rm -rf \
+		/tmp/* \
+		${HOME}/.cache
 
+# actual installation in extra layer so we can cache the stuff above
 
-COPY ./requirements.txt ./requirements.txt
+COPY --chown=abc:abc . .
 
 RUN \
-	apk add --no-cache --virtual .build-deps gcc g++ python3-dev libxml2-dev libxslt-dev libffi-dev libc-dev py3-pip linux-headers && \
-	pip install --no-cache-dir -r requirements.txt && \
-	apk del .build-deps
+	echo "**** install maloja ****" && \
+	apk add --no-cache --virtual=install-deps \
+	  py3-pip && \
+	pip3 install /usr/src/app && \
+	apk del --purge \
+		install-deps && \
+	rm -rf \
+		/tmp/* \
+		${HOME}/.cache
 
 
-# no chance for caching below here
 
-COPY . .
+COPY container/root/ /
 
-RUN pip install /usr/src/app
-
-# Docker-specific configuration
-# defaulting to IPv4 is no longer necessary (default host is dual stack)
-ENV MALOJA_SKIP_SETUP=yes
-ENV PYTHONUNBUFFERED=1
+ENV	\
+	# Docker-specific configuration
+	MALOJA_SKIP_SETUP=yes \
+	PYTHONUNBUFFERED=1 \
+	# Prevents breaking change for previous container that ran maloja as root
+	# On linux hosts (non-podman rootless) these variables should be set to the
+	# host user that should own the host folder bound to MALOJA_DATA_DIRECTORY
+	PUID=0 \
+	PGID=0
 
 EXPOSE 42010
-# use exec form for better signal handling https://docs.docker.com/engine/reference/builder/#entrypoint
-ENTRYPOINT ["maloja", "run"]

README.md

@@ -9,7 +9,7 @@
 
 Simple self-hosted music scrobble database to create personal listening statistics. No recommendations, no social network, no nonsense.
 
-![screenshot](screenshot.png?raw=true)
+![screenshot](https://raw.githubusercontent.com/krateng/maloja/master/screenshot.png)
 
 You can check [my own Maloja page](https://maloja.krateng.ch) as an example instance.
 
@@ -20,17 +20,13 @@ You can check [my own Maloja page](https://maloja.krateng.ch) as an example inst
 	* [Requirements](#requirements)
 	* [PyPI](#pypi)
 	* [From Source](#from-source)
-	* [Docker / Podman](#docker-podman)
+	* [Docker / Podman](#docker--podman)
 	* [Extras](#extras)
 * [How to use](#how-to-use)
 	* [Basic control](#basic-control)
 	* [Data](#data)
 	* [Customization](#customization)
 * [How to scrobble](#how-to-scrobble)
-	* [Native support](#native-support)
-	* [Native API](#native-api)
-	* [Standard-compliant API](#standard-compliant-api)
-	* [Manual](#manual)
 * [How to extend](#how-to-extend)
 
 ## Features
@@ -100,6 +96,23 @@ An example of a minimum run configuration to access maloja via `localhost:42010`
 	docker run -p 42010:42010 -v $PWD/malojadata:/mljdata -e MALOJA_DATA_DIRECTORY=/mljdata krateng/maloja
 ```
 
+#### Linux Host
+
+**NOTE:** If you are using [rootless containers with Podman](https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics#why_podman_) this DOES NOT apply to you.
+
+If you are running Docker on a **Linux Host** you should specify `user:group` ids of the user who owns the folder on the host machine bound to `MALOJA_DATA_DIRECTORY` in order to avoid [docker file permission problems.](https://ikriv.com/blog/?p=4698) These can be specified using the [environmental variables **PUID** and **PGID**.](https://docs.linuxserver.io/general/understanding-puid-and-pgid)
+
+To get the UID and GID for the current user run these commands from a terminal:
+
+* `id -u` -- prints UID (EX `1000`)
+* `id -g` -- prints GID (EX `1001`)
+
+The modified run command with these variables would look like:
+
+```console
+	docker run -e PUID=1000 -e PGID=1001 -p 42010:42010 -v $PWD/malojadata:/mljdata -e MALOJA_DATA_DIRECTORY=/mljdata krateng/maloja
+```
+
 ### Extras
 
 * If you'd like to display images, you will need API keys for [Last.fm](https://www.last.fm/api/account/create) and [Spotify](https://developer.spotify.com/dashboard/applications). These are free of charge!
@@ -139,6 +152,7 @@ If you would like to import your previous scrobbles, use the command `maloja imp
 
 * a Last.fm export generated by [benfoxall's website](https://benjaminbenben.com/lastfm-to-csv/) ([GitHub page](https://github.com/benfoxall/lastfm-to-csv))
 * an official [Spotify data export file](https://www.spotify.com/us/account/privacy/)
+* an official [ListenBrainz export file](https://listenbrainz.org/profile/export/)
 * the export of another Maloja instance
 
 ⚠️ Never import your data while maloja is running. When you need to do import inside docker container start it in shell mode instead and perform import before starting the container as mentioned above.

auxiliary/chromium_scrobbler/maloja-scrobbler/background.js

@@ -11,7 +11,8 @@ const ALWAYS_SCROBBLE_SECONDS = 60*3;
 // Longer songs are always scrobbled when playing at least 2 minutes
 
 pages = {
-	"Plex Web":{
+	"plex":{
+		"name":"Plex",
 		"patterns":[
 			"https://app.plex.tv",
 			"http://app.plex.tv",
@@ -20,31 +21,36 @@ pages = {
 		],
 		"script":"plex.js"
 	},
-	"YouTube Music":{
+	"ytmusic":{
+		"name":"YouTube Music",
 		"patterns":[
 			"https://music.youtube.com"
 		],
 		"script":"ytmusic.js"
 	},
-	"Spotify Web":{
+	"spotify":{
+		"name":"Spotify",
 		"patterns":[
 			"https://open.spotify.com"
 		],
 		"script":"spotify.js"
 	},
-	"Bandcamp":{
+	"bandcamp":{
+		"name":"Bandcamp",
 		"patterns":[
 			"bandcamp.com"
 		],
 		"script":"bandcamp.js"
 	},
-	"Soundcloud":{
+	"soundcloud":{
+		"name":"Soundcloud",
 		"patterns":[
 			"https://soundcloud.com"
 		],
 		"script":"soundcloud.js"
 	},
-	"Navidrome":{
+	"navidrome":{
+		"name":"Navidrome",
 		"patterns":[
 			"https://navidrome.",
 			"http://navidrome."
@@ -77,6 +83,13 @@ function onTabUpdated(tabId, changeInfo, tab) {
 				//console.log("Still on same page!")
 				tabManagers[tabId].update();
 
+				// check if the setting for this page is still active
+				chrome.storage.local.get(["service_active_" + page],function(result){
+					if (!result["service_active_" + page]) {
+						delete tabManagers[tabId];
+					}
+				});
+
 				return
 			}
 		}
@@ -90,13 +103,21 @@ function onTabUpdated(tabId, changeInfo, tab) {
 			patterns = pages[key]["patterns"];
 			for (var i=0;i<patterns.length;i++) {
 				if (tab.url.includes(patterns[i])) {
-					console.log("New page on tab " + tabId + " will be handled by new " + key + " manager!");
-					tabManagers[tabId] = new Controller(tabId,key);
-					updateTabNum();
-					return
-					//chrome.tabs.executeScript(tab.id,{"file":"sitescripts/" + pages[key]["script"]})
 
+					// check if we even like that page
+					chrome.storage.local.get(["service_active_" + key],function(result){
+						if (result["service_active_" + key]) {
+							console.log("New page on tab " + tabId + " will be handled by new " + key + " manager!");
+							tabManagers[tabId] = new Controller(tabId,key);
+							updateTabNum();
+							//chrome.tabs.executeScript(tab.id,{"file":"sitescripts/" + pages[key]["script"]})
+						}
+						else {
+							console.log("New page on tab " + tabId + " is " + key + ", not enabled!");
+						}
+					});
 
+					return;
 				}
 			}
 		}
@@ -127,10 +148,10 @@ function onInternalMessage(request,sender) {
 		for (tabId in tabManagers) {
 			manager = tabManagers[tabId]
 			if (manager.currentlyPlaying) {
-				answer.push([manager.page,manager.currentArtist,manager.currentTitle]);
+				answer.push([pages[manager.page]['name'],manager.currentArtist,manager.currentTitle]);
 			}
 			else {
-				answer.push([manager.page,null]);
+				answer.push([pages[manager.page]['name'],null]);
 			}
 
 		}

auxiliary/chromium_scrobbler/maloja-scrobbler/manifest.json

@@ -1,6 +1,6 @@
 {
 	"name": "Maloja Scrobbler",
-	"version": "1.11",
+	"version": "1.13",
 	"description": "Scrobbles tracks from various sites to your Maloja server",
 	"manifest_version": 2,
 	"permissions": [

auxiliary/chromium_scrobbler/maloja-scrobbler/settings.html

@@ -14,7 +14,7 @@
 				color:beige;
 				font-family:'Ubuntu';
 			}
-			input {
+			input[type=text] {
 				width:270px;
 				font-family:'Ubuntu';
 				outline:none;
@@ -33,10 +33,14 @@
 			<br /><br />
 			<span id="checkmark_key"></span> <span>API key:</span><br />
 			<input type="text" id="apikey" />
-			<br/><br/>
+			<hr/>
 			<span>Tabs:</span>
 			<list id="playinglist">
 			</list>
+			<hr/>
+			<span>Services:</span>
+			<list id="sitelist">
+			</list>
 		</div>
 	</body>
 </html>

auxiliary/chromium_scrobbler/maloja-scrobbler/settings.js

@@ -1,26 +1,71 @@
+// duplicate this info for now, don't know if there is a better way than sending messages
+var pages = {
+	"plex":"Plex",
+	"ytmusic":"YouTube Music",
+	"spotify":"Spotify",
+	"bandcamp":"Bandcamp",
+	"soundcloud":"Soundcloud",
+	"navidrome":"Navidrome"
+}
+
 var config_defaults = {
 	serverurl:"http://localhost:42010",
 	apikey:"BlackPinkInYourArea"
 }
 
+for (var key in pages) {
+	config_defaults["service_active_" + key] = true;
+}
+
 
 document.addEventListener("DOMContentLoaded",function() {
 
+	var sitelist = document.getElementById("sitelist");
+
+
+	for (var identifier in pages) {
+		sitelist.append(document.createElement('br'));
+		var checkbox = document.createElement('input');
+		checkbox.type = "checkbox";
+		checkbox.id = "service_active_" + identifier;
+		var label = document.createElement('label');
+		label.for = checkbox.id;
+		label.textContent = pages[identifier];
+		sitelist.appendChild(checkbox);
+		sitelist.appendChild(label);
+
+		checkbox.addEventListener("change",toggleSite);
+
+	}
+
+
+
 	document.getElementById("serverurl").addEventListener("change",checkServer);
 	document.getElementById("apikey").addEventListener("change",checkServer);
 
 	document.getElementById("serverurl").addEventListener("focusout",checkServer);
 	document.getElementById("apikey").addEventListener("focusout",checkServer);
 
-	document.getElementById("serverurl").addEventListener("input",saveConfig);
-	document.getElementById("apikey").addEventListener("input",saveConfig);
+	document.getElementById("serverurl").addEventListener("input",saveServer);
+	document.getElementById("apikey").addEventListener("input",saveServer);
 
 
 	chrome.runtime.onMessage.addListener(onInternalMessage);
 
 	chrome.storage.local.get(config_defaults,function(result){
+		console.log(result);
 		for (var key in result) {
-			document.getElementById(key).value = result[key];
+
+			// booleans
+			if (result[key] == true || result[key] == false) {
+				document.getElementById(key).checked = result[key];
+			}
+
+			// text
+			else{
+				document.getElementById(key).value = result[key];
+			}
+
 		}
 		checkServer();
 	})
@@ -31,6 +76,11 @@ document.addEventListener("DOMContentLoaded",function() {
 
 });
 
+function toggleSite(evt) {
+	var element = evt.target;
+	chrome.storage.local.set({ [element.id]: element.checked });
+}
+
 
 function onInternalMessage(request,sender) {
 	if (request.type == "response") {
@@ -50,8 +100,8 @@ function onInternalMessage(request,sender) {
 
 
 
-function saveConfig() {
-	for (var key in config_defaults) {
+function saveServer() {
+	for (var key of ["serverurl","apikey"]) {
 		var value = document.getElementById(key).value;
 		chrome.storage.local.set({ [key]: value });
 	}

container/root/etc/s6-overlay/s6-rc.d/init-permission-check/run

@@ -0,0 +1,10 @@
+#!/usr/bin/with-contenv bash
+
+if [ "$(s6-setuidgid abc id -u)" = "0" ]; then
+   echo "-------------------------------------"
+   echo "WARN: Running as root! If you meant to do this than this message can be ignored."
+   echo "If you are running this container on a *linux* host and are not using podman rootless you SHOULD"
+   echo "change the ENVs PUID and PGID for this container to ensure correct permissions on your config folder."
+   echo -e "See: https://github.com/krateng/maloja#linux-host\n"
+   echo -e "-------------------------------------\n"
+fi

container/root/etc/s6-overlay/s6-rc.d/init-permission-check/type

@@ -0,0 +1 @@
+oneshot

container/root/etc/s6-overlay/s6-rc.d/init-permission-check/up

@@ -0,0 +1 @@
+/etc/s6-overlay/s6-rc.d/init-permission-check/run

container/root/etc/s6-overlay/s6-rc.d/svc-python/run

@@ -0,0 +1,7 @@
+#!/usr/bin/with-contenv bash
+
+# used https://github.com/linuxserver/docker-wikijs/blob/master/root/etc/s6-overlay/s6-rc.d/svc-wikijs/run as a template
+
+echo -e "\nMaloja is starting!"
+exec \
+	  s6-setuidgid abc python -m maloja run

container/root/etc/s6-overlay/s6-rc.d/svc-python/type

@@ -0,0 +1 @@
+longrun

dev/list_tags.sh

@@ -0,0 +1 @@
+git tag -l '*.0' -n1 --sort=v:refname

dev/releases/3.1.yml

@@ -11,6 +11,36 @@ minor_release_name: "Soyeon"
     - "[Logging] Added cache memory use information"
     - "[Technical] Bumped Python Version and various dependencies"
 3.1.1:
+    commit: "20aae955b2263be07c56bafe4794f622117116ef"
     notes:
     - "[Bugfix] Fixed inclusion of custom css files"
     - "[Bugfix] Fixed list values in configuration"
+3.1.2:
+    commit: "a0739306013cd9661f028fb5b2620cfa2d298aa4"
+    notes:
+    - "[Feature] Added remix artist parsing"
+    - "[Feature] Added API debug mode"
+    - "[Bugfix] Fixed leftover whitespaces when parsing titles"
+    - "[Bugfix] Fixed handling of fallthrough values in config file"
+3.1.3:
+    commit: "f3a04c79b1c37597cdf3cafcd95e3c923cd6a53f"
+    notes:
+    - "[Bugfix] Fixed infinite recursion with capitalized featuring delimiters"
+    - "[Bugfix] Fixed favicon display"
+3.1.4:
+    commit: "ef06f2262205c903e7c3060e2d2d52397f8ffc9d"
+    notes:
+    - "[Feature] Expanded information saved from Listenbrainz API"
+    - "[Feature] Added import for Listenbrainz exports"
+    - "[Bugfix] Sanitized artists and tracks with html-like structure"
+3.1.5:
+    commit: "4330b0294bc0a01cdb841e2e3db370108da901db"
+    notes:
+    - "[Feature] Made image upload part of regular API"
+    - "[Bugfix] Additional entity name sanitization"
+    - "[Bugfix] Fixed image display on Safari"
+    - "[Bugfix] Fixed entity editing on Firefox"
+    - "[Bugfix] Made compatibile with SQLAlchemy 2.0"
+upcoming:
+    notes:
+    - "[Bugfix] Fixed configuration of time format"

maloja/__main__.py

@@ -14,7 +14,7 @@ from . import __pkginfo__ as pkginfo
 from .pkg_global import conf
 from .proccontrol import tasks
 from .setup import setup
-from .dev import generate
+from .dev import generate, apidebug
 
 
 
@@ -137,9 +137,15 @@ def print_info():
 	print(col['lightblue']("Configuration Directory:"),conf.dir_settings['config'])
 	print(col['lightblue']("Data Directory:         "),conf.dir_settings['state'])
 	print(col['lightblue']("Log Directory:          "),conf.dir_settings['logs'])
-	print(col['lightblue']("Network:                "),f"IPv{ip_address(conf.malojaconfig['host']).version}, Port {conf.malojaconfig['port']}")
+	print(col['lightblue']("Network:                "),f"Dual Stack, Port {conf.malojaconfig['port']}" if conf.malojaconfig['host'] == "*" else f"IPv{ip_address(conf.malojaconfig['host']).version}, Port {conf.malojaconfig['port']}")
 	print(col['lightblue']("Timezone:               "),f"UTC{conf.malojaconfig['timezone']:+d}")
 	print()
+	try:
+		import pkg_resources
+		for pkg in ("sqlalchemy","waitress","bottle","doreah","jinja2"):
+			print(col['cyan']     (f"{pkg}:".ljust(13)),pkg_resources.get_distribution(pkg).version)
+	except ImportError:
+		print("Could not determine dependency versions.")
 	print()
 
 @mainfunction({"l":"level","v":"version","V":"version"},flags=['version','include_images'],shield=True)
@@ -159,6 +165,7 @@ def main(*args,**kwargs):
 		"backup":tasks.backup,					# maloja backup --targetfolder /x/y --include_images
 		"generate":generate.generate_scrobbles,	# maloja generate 400
 		"export":tasks.export,					# maloja export
+		"apidebug":apidebug.run,				# maloja apidebug
 		# aux
 		"info":print_info
 	}

maloja/__pkginfo__.py

@@ -4,7 +4,7 @@
 # you know what f*ck it
 # this is hardcoded for now because of that damn project / package name discrepancy
 # i'll fix it one day
-VERSION = "3.1.1"
+VERSION = "3.1.5"
 HOMEPAGE = "https://github.com/krateng/maloja"
 
 

maloja/apis/__init__.py

@@ -47,9 +47,12 @@ def init_apis(server):
 			server.get(altpath_empty_cl)(alias_api)
 			server.post(altpath_empty_cl)(alias_api)
 
-	def invalid_api(pth):
+	def invalid_api(pth=''):
 		response.status = 404
 		return {"error":"Invalid API"}
 
 	server.get("/apis/<pth:path>")(invalid_api)
 	server.post("/apis/<pth:path>")(invalid_api)
+
+	server.get("/apis")(invalid_api)
+	server.post("/apis")(invalid_api)

maloja/apis/audioscrobbler_legacy.py

@@ -73,6 +73,8 @@ class AudioscrobblerLegacy(APIHandler):
 		client = self.mobile_sessions.get(key)
 		for count in range(50):
 			artist_key = f"a[{count}]"
+			album_key = f"b[{count}]"
+			length_key = f"l[{count}]"
 			track_key = f"t[{count}]"
 			time_key = f"i[{count}]"
 			if artist_key not in keys or track_key not in keys:
@@ -82,12 +84,19 @@ class AudioscrobblerLegacy(APIHandler):
 				timestamp = int(keys[time_key])
 			except Exception:
 				timestamp = None
-			#database.createScrobble(artists,title,timestamp)
-			self.scrobble({
+
+			scrobble = {
 				'track_artists':[artiststr],
 				'track_title':titlestr,
-				'scrobble_time':timestamp
-			},client=client)
+				'scrobble_time':timestamp,
+			}
+			if album_key in keys:
+				scrobble['album_name'] = keys[album_key]
+			if length_key in keys:
+				scrobble['track_length'] = keys[length_key]
+
+			#database.createScrobble(artists,title,timestamp)
+			self.scrobble(scrobble, client=client)
 		return 200,"OK\n"
 
 

maloja/apis/listenbrainz.py

@@ -55,6 +55,8 @@ class Listenbrainz(APIHandler):
 				try:
 					metadata = listen["track_metadata"]
 					artiststr, titlestr = metadata["artist_name"], metadata["track_name"]
+					albumstr = metadata.get("release_name")
+					additional = metadata.get("additional_info",{})
 					try:
 						timestamp = int(listen["listened_at"])
 					except Exception:
@@ -62,10 +64,21 @@ class Listenbrainz(APIHandler):
 				except Exception:
 					raise MalformedJSONException()
 
+				extrafields = {
+					# fields that will not be consumed by regular scrobbling
+					# will go into 'extra'
+					k:additional[k]
+					for k in ['track_mbid', 'release_mbid', 'artist_mbids','recording_mbid','tags']
+					if k in additional
+				}
+
 				self.scrobble({
 					'track_artists':[artiststr],
 					'track_title':titlestr,
-					'scrobble_time':timestamp
+					'album_name':albumstr,
+					'scrobble_time':timestamp,
+					'track_length': additional.get("duration"),
+					**extrafields
 				},client=client)
 
 			return 200,{"status":"ok"}

maloja/apis/native_v1.py

@@ -72,6 +72,14 @@ errors = {
 			'desc':"The database is being upgraded. Please try again later."
 		}
 	}),
+	images.MalformedB64: lambda e: (400,{
+		"status":"failure",
+		"error":{
+			'type':'malformed_b64',
+			'value':None,
+			'desc':"The provided base 64 string is not valid."
+		}
+	}),
 	# for http errors, use their status code
 	Exception: lambda e: ((e.status_code if hasattr(e,'statuscode') else 500),{
 		"status":"failure",
@@ -504,6 +512,32 @@ def post_scrobble(
 
 
 
+@api.post("addpicture")
+@authenticated_function(alternate=api_key_correct,api=True)
+@catch_exceptions
+def add_picture(b64,artist:Multi=[],title=None):
+	"""Uploads a new image for an artist or track.
+
+	param string b64: Base 64 representation of the image
+	param string artist: Artist name. Can be supplied multiple times for tracks with multiple artists.
+	param string title: Title of the track. Optional.
+
+	"""
+	keys = FormsDict()
+	for a in artist:
+		keys.append("artist",a)
+	if title is not None: keys.append("title",title)
+	k_filter, _, _, _, _ = uri_to_internal(keys)
+	if "track" in k_filter: k_filter = k_filter["track"]
+	url = images.set_image(b64,**k_filter)
+
+	return {
+		'status': 'success',
+		'url': url
+	}
+
+
+
 @api.post("importrules")
 @authenticated_function(api=True)
 @catch_exceptions
@@ -582,20 +616,6 @@ def search(**keys):
 	return {"artists":artists_result[:max_],"tracks":tracks_result[:max_]}
 
 
-@api.post("addpicture")
-@authenticated_function(api=True)
-@catch_exceptions
-def add_picture(b64,artist:Multi=[],title=None):
-	"""Internal Use Only"""
-	keys = FormsDict()
-	for a in artist:
-		keys.append("artist",a)
-	if title is not None: keys.append("title",title)
-	k_filter, _, _, _, _ = uri_to_internal(keys)
-	if "track" in k_filter: k_filter = k_filter["track"]
-	images.set_image(b64,**k_filter)
-
-
 @api.post("newrule")
 @authenticated_function(api=True)
 @catch_exceptions

maloja/cleanup.py

@@ -55,7 +55,7 @@ class CleanerAgent:
 		artists = list(set(artists))
 		artists.sort()
 
-		return (artists,title)
+		return (artists,title.strip())
 
 	def removespecial(self,s):
 		if isinstance(s,list):
@@ -82,7 +82,7 @@ class CleanerAgent:
 
 	def parseArtists(self,a):
 
-		if isinstance(a,list):
+		if isinstance(a,list) or isinstance(a,tuple):
 			res = [self.parseArtists(art) for art in a]
 			return [a for group in res for a in group]
 
@@ -109,9 +109,9 @@ class CleanerAgent:
 
 
 		for d in self.delimiters_feat:
-			if re.match(r"(.*) [\(\[]" + d + " (.*)[\)\]]",a) is not None:
-				return self.parseArtists(re.sub(r"(.*) [\(\[]" + d + " (.*)[\)\]]",r"\1",a)) + \
-						self.parseArtists(re.sub(r"(.*) [\(\[]" + d + " (.*)[\)\]]",r"\2",a))
+			if re.match(r"(.*) [\(\[]" + d + " (.*)[\)\]]",a,flags=re.IGNORECASE) is not None:
+				return self.parseArtists(re.sub(r"(.*) [\(\[]" + d + " (.*)[\)\]]",r"\1",a,flags=re.IGNORECASE)) + \
+						self.parseArtists(re.sub(r"(.*) [\(\[]" + d + " (.*)[\)\]]",r"\2",a,flags=re.IGNORECASE))
 
 
 
@@ -156,25 +156,37 @@ class CleanerAgent:
 		#	t = p(t).strip()
 		return t
 
-	def parseTitleForArtists(self,t):
-		for d in self.delimiters_feat:
-			if re.match(r"(.*) [\(\[]" + d + " (.*?)[\)\]]",t) is not None:
-				(title,artists) = self.parseTitleForArtists(re.sub(r"(.*) [\(\[]" + d + " (.*?)[\)\]]",r"\1",t))
-				artists += self.parseArtists(re.sub(r"(.*) [\(\[]" + d + " (.*?)[\)\]].*",r"\2",t))
-				return (title,artists)
-			if re.match(r"(.*) - " + d + " (.*)",t) is not None:
-				(title,artists) = self.parseTitleForArtists(re.sub(r"(.*) - " + d + " (.*)",r"\1",t))
-				artists += self.parseArtists(re.sub(r"(.*) - " + d + " (.*).*",r"\2",t))
-				return (title,artists)
-			if re.match(r"(.*) " + d + " (.*)",t) is not None:
-				(title,artists) = self.parseTitleForArtists(re.sub(r"(.*) " + d + " (.*)",r"\1",t))
-				artists += self.parseArtists(re.sub(r"(.*) " + d + " (.*).*",r"\2",t))
-				return (title,artists)
-
+	def parseTitleForArtists(self,title):
 		artists = []
+		for delimiter in malojaconfig["DELIMITERS_FEAT"]:
+			for pattern in [
+				r" [\(\[]" + re.escape(delimiter) + " (.*?)[\)\]]",
+				r" - " + re.escape(delimiter) + " (.*)",
+				r" " + re.escape(delimiter) + " (.*)"
+			]:
+				matches = re.finditer(pattern,title,flags=re.IGNORECASE)
+				for match in matches:
+					title = match.re.sub('',match.string) # Remove matched part
+					artists += self.parseArtists(match.group(1)) # Parse matched artist string
+
+
+
+		if malojaconfig["PARSE_REMIX_ARTISTS"]:
+			for filter in malojaconfig["FILTERS_REMIX"]:
+				for pattern in [
+					r" [\(\[](.*)" + re.escape(filter) + "[\)\]]", # match remix in brackets
+					r" - (.*)" + re.escape(filter) # match remix split with "-"
+				]:
+					match = re.search(pattern,title,flags=re.IGNORECASE)
+					if match:
+						# title stays the same
+						artists += self.parseArtists(match.group(1))
+
+
+
 		for st in self.rules_artistintitle:
-			if st in t.lower(): artists += self.rules_artistintitle[st].split("␟")
-		return (t,artists)
+			if st in title.lower(): artists += self.rules_artistintitle[st].split("␟")
+		return (title,artists)
 
 
 

maloja/data_files/config/rules/predefined/krateng_artistsingroups.tsv

@@ -8,6 +8,7 @@ countas		Trouble Maker				HyunA
 countas		S Club 7				Tina Barrett
 countas		4Minute						HyunA
 countas		I.O.I						Chungha
+countas		TrySail					Sora Amamiya
 # Group more famous than single artist
 countas		RenoakRhythm				Approaching Nirvana
 countas		Shirley Manson				Garbage
@@ -18,3 +19,7 @@ countas		Airi Suzuki					℃-ute
 countas		CeeLo Green					Gnarls Barkley
 countas		Amelia Watson				Hololive EN
 countas		Gawr Gura					Hololive EN
+countas		Mori Calliope				Hololive EN
+countas		Ninomae Ina'nis				Hololive EN
+countas		Takanashi Kiara				Hololive EN
+countas		Ceres Fauna						Hololive EN

maloja/data_files/config/rules/predefined/krateng_jpop.tsv

@@ -0,0 +1,20 @@
+# NAME: JPop
+# DESC: Fixes and romanizes various Japanese tracks and artists
+
+
+belongtogether	Myth & Roid
+
+
+# Sora-chan
+replaceartist	Amamiya Sora						Sora Amamiya
+replacetitle	エデンの旅人						Eden no Tabibito
+replacetitle	月灯り							Tsukiakari
+replacetitle	火花							Hibana
+replacetitle	ロンリーナイト・ディスコティック				Lonely Night Discotheque
+replacetitle	羽根輪舞							Hane Rinbu
+replacetitle	メリーゴーランド						Merry-go-round
+replacetitle	フリイジア						Fressia
+replacetitle	誓い							Chikai
+
+# ReoNa
+replacetitle	ないない							nainai

maloja/data_files/config/rules/predefined/krateng_kpopgirlgroups.tsv

@@ -21,7 +21,7 @@ addartists		HyunA					Change		Jun Hyung
 # BLACKPINK
 countas		Jennie					BLACKPINK
 countas		Rosé					BLACKPINK
-countas		Lisa					BLACKPINK
+countas		Lalisa					BLACKPINK
 countas		Jisoo					BLACKPINK
 replacetitle	AS IF IT'S YOUR LAST			As If It's Your Last
 replacetitle	BOOMBAYAH				Boombayah
@@ -200,10 +200,13 @@ countas				ACE					IZ*ONE
 countas				Chaewon			IZ*ONE
 countas				Minju			IZ*ONE
 
-
+# ITZY
 countas				Yeji			ITZY
+
+# IVE
 countas				Wonyoung		IVE
 countas				Yujin			IVE
+countas				Gaeul			IVE
 
 # Popular Remixes
 artistintitle	Areia Remix				Areia

maloja/database/__init__.py

@@ -148,7 +148,7 @@ def rawscrobble_to_scrobbledict(rawscrobble, fix=True, client=None):
 		"origin":f"client:{client}" if client else "generic",
 		"extra":{
 			k:scrobbleinfo[k] for k in scrobbleinfo if k not in
-			['scrobble_time','track_artists','track_title','track_length','scrobble_duration','album_name','album_artists']
+			['scrobble_time','track_artists','track_title','track_length','scrobble_duration']#,'album_name','album_artists']
 		},
 		"rawscrobble":rawscrobble
 	}

maloja/database/sqldb.py

@@ -298,7 +298,7 @@ def get_track_id(trackdict,create_new=True,dbconn=None):
 
 
 	op = DB['tracks'].select(
-		DB['tracks'].c.id
+#		DB['tracks'].c.id
 	).where(
 		DB['tracks'].c.title_normalized==ntitle
 	)
@@ -308,7 +308,7 @@ def get_track_id(trackdict,create_new=True,dbconn=None):
 		foundtrackartists = []
 
 		op = DB['trackartists'].select(
-			DB['trackartists'].c.artist_id
+#			DB['trackartists'].c.artist_id
 		).where(
 			DB['trackartists'].c.track_id==row[0]
 		)
@@ -344,7 +344,7 @@ def get_artist_id(artistname,create_new=True,dbconn=None):
 	#print("looking for",nname)
 
 	op = DB['artists'].select(
-		DB['artists'].c.id
+#		DB['artists'].c.id
 	).where(
 		DB['artists'].c.name_normalized==nname
 	)

maloja/dev/apidebug.py

@@ -0,0 +1,28 @@
+import bottle, waitress
+
+from ..pkg_global.conf import malojaconfig
+
+from doreah.logging import log
+from nimrodel import EAPI as API
+
+
+PORT = malojaconfig["PORT"]
+HOST = malojaconfig["HOST"]
+
+the_listener = API(delay=True)
+
+@the_listener.get("{path}")
+@the_listener.post("{path}")
+def all_requests(path,**kwargs):
+	result = {
+		'path':path,
+		'payload': kwargs
+	}
+	log(result)
+	return result
+
+
+def run():
+	server = bottle.Bottle()
+	the_listener.mount(server,path="apis")
+	waitress.serve(server, listen=f"*:{PORT}")

maloja/images.py

@@ -267,6 +267,9 @@ def local_files(artist=None,artists=None,title=None):
 
 
 
+class MalformedB64(Exception):
+	pass
+
 def set_image(b64,**keys):
 	track = "title" in keys
 	if track:
@@ -279,7 +282,10 @@ def set_image(b64,**keys):
 	log("Trying to set image, b64 string: " + str(b64[:30] + "..."),module="debug")
 
 	regex = r"data:image/(\w+);base64,(.+)"
-	type,b64 = re.fullmatch(regex,b64).groups()
+	match = re.fullmatch(regex,b64)
+	if not match: raise MalformedB64()
+
+	type,b64 = match.groups()
 	b64 = base64.b64decode(b64)
 	filename = "webupload" + str(int(datetime.datetime.now().timestamp())) + "." + type
 	for folder in get_all_possible_filenames(**keys):
@@ -293,8 +299,11 @@ def set_image(b64,**keys):
 		with open(data_dir['images'](folder,filename),"wb") as f:
 			f.write(b64)
 
+
 	log("Saved image as " + data_dir['images'](folder,filename),module="debug")
 
 	# set as current picture in rotation
 	if track: set_image_in_cache(id,'tracks',os.path.join("/images",folder,filename))
 	else: set_image_in_cache(id,'artists',os.path.join("/images",folder,filename))
+
+	return os.path.join("/images",folder,filename)

maloja/pkg_global/conf.py

@@ -179,15 +179,18 @@ malojaconfig = Configuration(
 		"Database":{
 			"invalid_artists":(tp.Set(tp.String()),								"Invalid Artists",				["[Unknown Artist]","Unknown Artist","Spotify"],											"Artists that should be discarded immediately"),
 			"remove_from_title":(tp.Set(tp.String()),							"Remove from Title",			["(Original Mix)","(Radio Edit)","(Album Version)","(Explicit Version)","(Bonus Track)"],	"Phrases that should be removed from song titles"),
-			"delimiters_feat":(tp.Set(tp.String()),								"Featuring Delimiters",			["ft.","ft","feat.","feat","featuring","Ft.","Ft","Feat.","Feat","Featuring"],				"Delimiters used for extra artists, even when in the title field"),
+			"delimiters_feat":(tp.Set(tp.String()),								"Featuring Delimiters",			["ft.","ft","feat.","feat","featuring"],													"Delimiters used for extra artists, even when in the title field"),
 			"delimiters_informal":(tp.Set(tp.String()),							"Informal Delimiters",			["vs.","vs","&"],																			"Delimiters in informal artist strings with spaces expected around them"),
-			"delimiters_formal":(tp.Set(tp.String()),							"Formal Delimiters",			[";","/","|","␝","␞","␟"],																					"Delimiters used to tag multiple artists when only one tag field is available")
+			"delimiters_formal":(tp.Set(tp.String()),							"Formal Delimiters",			[";","/","|","␝","␞","␟"],																	"Delimiters used to tag multiple artists when only one tag field is available"),
+			"filters_remix":(tp.Set(tp.String()),								"Remix Filters",				["Remix", "Remix Edit", "Short Mix", "Extended Mix", "Soundtrack Version"],					"Filters used to recognize the remix artists in the title"),
+			"parse_remix_artists":(tp.Boolean(),								"Parse Remix Artists",			False)
 		},
 		"Web Interface":{
 			"default_range_charts_artists":(tp.Choice({'alltime':'All Time','year':'Year','month':"Month",'week':'Week'}),	"Default Range Artist Charts",	"year"),
 			"default_range_charts_tracks":(tp.Choice({'alltime':'All Time','year':'Year','month':"Month",'week':'Week'}),	"Default Range Track Charts",	"year"),
 			"default_step_pulse":(tp.Choice({'year':'Year','month':"Month",'week':'Week','day':'Day'}),						"Default Pulse Step",			"month"),
 			"charts_display_tiles":(tp.Boolean(),								"Display Chart Tiles",			False),
+			"display_art_icons":(tp.Boolean(),									"Display Album/Artist Icons",	True),
 			"discourage_cpu_heavy_stats":(tp.Boolean(),							"Discourage CPU-heavy stats",	False,					"Prevent visitors from mindlessly clicking on CPU-heavy options. Does not actually disable them for malicious actors!"),
 			"use_local_images":(tp.Boolean(),									"Use Local Images",				True),
 			#"local_image_rotate":(tp.Integer(),									"Local Image Rotate",			3600),

maloja/proccontrol/tasks/import_scrobbles.py

@@ -37,18 +37,27 @@ def import_scrobbles(inputf):
 		typeid,typedesc = "lastfm","Last.fm"
 		importfunc = parse_lastfm
 
+	elif re.match("Streaming_History_Audio.+\.json",filename):
+		typeid,typedesc = "spotify","Spotify"
+		importfunc = parse_spotify_lite
+
 	elif re.match("endsong_[0-9]+\.json",filename):
 		typeid,typedesc = "spotify","Spotify"
-		importfunc = parse_spotify_full
+		importfunc = parse_spotify
 
 	elif re.match("StreamingHistory[0-9]+\.json",filename):
 		typeid,typedesc = "spotify","Spotify"
-		importfunc = parse_spotify_lite
+		importfunc = parse_spotify_lite_legacy
 
 	elif re.match("maloja_export_[0-9]+\.json",filename):
 		typeid,typedesc = "maloja","Maloja"
 		importfunc = parse_maloja
 
+	# username_lb-YYYY-MM-DD.json
+	elif re.match(".*_lb-[0-9-]+\.json",filename):
+		typeid,typedesc = "listenbrainz","ListenBrainz"
+		importfunc = parse_listenbrainz
+
 	else:
 		print("File",inputf,"could not be identified as a valid import source.")
 		return result
@@ -76,6 +85,7 @@ def import_scrobbles(inputf):
 			# extra info
 			extrainfo = {}
 			if scrobble.get('album_name'): extrainfo['album_name'] = scrobble['album_name']
+			if scrobble.get('album_artist'): extrainfo['album_artist'] = scrobble['album_artist']
 			# saving this in the scrobble instead of the track because for now it's not meant
 			# to be authorative information, just payload of the scrobble
 
@@ -84,7 +94,7 @@ def import_scrobbles(inputf):
 				 	"track":{
 				 		"artists":scrobble['track_artists'],
 				 		"title":scrobble['track_title'],
-				 		"length":None
+				 		"length":scrobble['track_length'],
 				 	},
 				 	"duration":scrobble['scrobble_duration'],
 				 	"origin":"import:" + typeid,
@@ -116,7 +126,7 @@ def import_scrobbles(inputf):
 
 	return result
 
-def parse_spotify_lite(inputf):
+def parse_spotify_lite_legacy(inputf):
 	pth = os.path
 	inputfolder = pth.relpath(pth.dirname(pth.abspath(inputf)))
 	filenames = re.compile(r'StreamingHistory[0-9]+\.json')
@@ -154,6 +164,7 @@ def parse_spotify_lite(inputf):
 				yield ("CONFIDENT_IMPORT",{
 					'track_title':title,
 					'track_artists': artist,
+					'track_length': None,
 					'scrobble_time': timestamp,
 					'scrobble_duration':played,
 					'album_name': None
@@ -165,7 +176,59 @@ def parse_spotify_lite(inputf):
 		print()
 
 
-def parse_spotify_full(inputf):
+def parse_spotify_lite(inputf):
+	pth = os.path
+	inputfolder = pth.relpath(pth.dirname(pth.abspath(inputf)))
+	filenames = re.compile(r'Streaming_History_Audio.+\.json')
+	inputfiles = [os.path.join(inputfolder,f) for f in os.listdir(inputfolder) if filenames.match(f)]
+
+	if len(inputfiles) == 0:
+		print("No files found!")
+		return
+
+	if inputfiles != [inputf]:
+		print("Spotify files should all be imported together to identify duplicates across the whole dataset.")
+		if not ask("Import " + ", ".join(col['yellow'](i) for i in inputfiles) + "?",default=True):
+			inputfiles = [inputf]
+
+	for inputf in inputfiles:
+
+		print("Importing",col['yellow'](inputf),"...")
+		with open(inputf,'r') as inputfd:
+			data = json.load(inputfd)
+
+		for entry in data:
+
+			try:
+				played = int(entry['ms_played'] / 1000)
+				timestamp = int(
+					datetime.datetime.strptime(entry['ts'],"%Y-%m-%dT%H:%M:%SZ").timestamp()
+				)
+				artist = entry['master_metadata_album_artist_name'] # hmmm
+				title = entry['master_metadata_track_name']
+				album = entry['master_metadata_album_album_name']
+				albumartist = entry['master_metadata_album_artist_name']
+
+				if played < 30:
+					yield ('CONFIDENT_SKIP',None,f"{entry} is shorter than 30 seconds, skipping...")
+					continue
+
+				yield ("CONFIDENT_IMPORT",{
+					'track_title':title,
+					'track_artists': artist,
+					'track_length': None,
+					'scrobble_time': timestamp,
+					'scrobble_duration':played,
+					'album_name': album,
+					'album_artist': albumartist
+				},'')
+			except Exception as e:
+				yield ('FAIL',None,f"{entry} could not be parsed. Scrobble not imported. ({repr(e)})")
+				continue
+
+		print()
+
+def parse_spotify(inputf):
 	pth = os.path
 	inputfolder = pth.relpath(pth.dirname(pth.abspath(inputf)))
 	filenames = re.compile(r'endsong_[0-9]+\.json')
@@ -174,7 +237,7 @@ def parse_spotify_full(inputf):
 	if len(inputfiles) == 0:
 		print("No files found!")
 		return
-		
+
 	if inputfiles != [inputf]:
 		print("Spotify files should all be imported together to identify duplicates across the whole dataset.")
 		if not ask("Import " + ", ".join(col['yellow'](i) for i in inputfiles) + "?",default=True):
@@ -262,6 +325,7 @@ def parse_spotify_full(inputf):
 				yield (status,{
 					'track_title':title,
 					'track_artists': artist,
+					'track_length': None,
 					'album_name': album,
 					'scrobble_time': timestamp,
 					'scrobble_duration':played
@@ -294,6 +358,7 @@ def parse_lastfm(inputf):
 				yield ('CONFIDENT_IMPORT',{
 					'track_title': title,
 					'track_artists': artist,
+					'track_length': None,
 					'album_name': album,
 					'scrobble_time': int(datetime.datetime.strptime(
 						time + '+0000',
@@ -305,6 +370,28 @@ def parse_lastfm(inputf):
 				yield ('FAIL',None,f"{row} (Line {line}) could not be parsed. Scrobble not imported. ({repr(e)})")
 				continue
 
+def parse_listenbrainz(inputf):
+
+	with open(inputf,'r') as inputfd:
+		data = json.load(inputfd)
+
+	for entry in data:
+
+		try:
+			track_metadata = entry['track_metadata']
+			additional_info = track_metadata.get('additional_info', {})
+
+			yield ("CONFIDENT_IMPORT",{
+				'track_title': track_metadata['track_name'],
+				'track_artists': additional_info.get('artist_names') or track_metadata['artist_name'],
+				'track_length': int(additional_info.get('duration_ms', 0) / 1000) or additional_info.get('duration'),
+				'album_name': track_metadata.get('release_name'),
+				'scrobble_time': entry['listened_at'],
+				'scrobble_duration': None,
+			},'')
+		except Exception as e:
+			yield ('FAIL',None,f"{entry} could not be parsed. Scrobble not imported. ({repr(e)})")
+			continue
 
 def parse_maloja(inputf):
 
@@ -318,6 +405,7 @@ def parse_maloja(inputf):
 			yield ('CONFIDENT_IMPORT',{
 				'track_title': s['track']['title'],
 				'track_artists': s['track']['artists'],
+				'track_length': s['track']['length'],
 				'album_name': s['track'].get('album',{}).get('name',''),
 				'scrobble_time': s['time'],
 				'scrobble_duration': s['duration']

maloja/server.py

@@ -10,6 +10,7 @@ import time
 # server stuff
 from bottle import Bottle, static_file, request, response, FormsDict, redirect, BaseRequest, abort
 import waitress
+from jinja2.exceptions import TemplateNotFound
 
 # doreah toolkit
 from doreah.logging import log
@@ -212,10 +213,11 @@ def jinja_page(name):
 			"_urikeys":keys, #temporary!
 		}
 		loc_context["filterkeys"], loc_context["limitkeys"], loc_context["delimitkeys"], loc_context["amountkeys"], loc_context["specialkeys"] = uri_to_internal(keys)
-
-		template = jinja_environment.get_template(name + '.jinja')
 		try:
+			template = jinja_environment.get_template(name + '.jinja')
 			res = template.render(**loc_context)
+		except TemplateNotFound:
+			abort(404,f"Not found: '{name}'")
 		except (ValueError, IndexError):
 			abort(404,"This Artist or Track does not exist")
 

maloja/thirdparty/musicbrainz.py

@@ -18,7 +18,7 @@ class MusicBrainz(MetadataInterface):
 
 	metadata = {
 		"response_type":"json",
-		"response_parse_tree_track": ["images",0,"image"],
+		"response_parse_tree_track": ["images",0,"thumbnails","500"],
 		"required_settings": [],
 	}
 

maloja/web/jinja/abstracts/base.jinja

@@ -8,6 +8,7 @@
 
 		<title>{% block title %}{% endblock %}</title>
 		<meta name="description" content='Maloja is a self-hosted music scrobble server.' />
+		<link rel="icon" type="image/x-icon" href="/favicon.ico" />
 
 		<meta name="color-scheme" content="dark" />
 		<meta name="darkreader" content="wat" />
@@ -96,5 +97,10 @@
 		</div>
 
 
+    <!-- Load script as late as possible so DOM renders first -->
+    <script src="/lazyload17-8-2.min.js"></script>
+    <script>
+        var lazyLoadInstance = new LazyLoad({});
+    </script>
 	</body>
 </html>

maloja/web/jinja/admin_manual.jinja

@@ -16,7 +16,7 @@
 			<td style="padding-right:7px;">
 				Artists:
 			</td><td id="artists_td">
-				<input placeholder='Separate with Enter' class='simpleinput' id='artists' onKeydown='keyDetect(event)' />
+				<input placeholder='Separate with Enter' class='simpleinput' id='artists' onKeydown='keyDetect(event)' onblur='addEnteredArtist()' />
 			</td>
 		</tr>
 		<tr>

maloja/web/jinja/admin_overview.jinja

@@ -83,10 +83,10 @@
 
 	Backup your data.<br/><br/>
 
-	<a href="/apis/mlj_1/backup" download="maloja_backup.tar.gz">
+	<a class="hidelink" href="/apis/mlj_1/backup" download="maloja_backup.tar.gz">
 		<button type="button">Backup</button>
 	</a>
-	<a href="/apis/mlj_1/export" download="maloja_export.json">
+	<a class="hidelink" href="/apis/mlj_1/export" download="maloja_export.json">
 		<button type="button">Export</button>
 	</a>
 

maloja/web/jinja/admin_setup.jinja

@@ -90,7 +90,7 @@
 
 	<h2>Set up some rules</h2>
 
-	After you've scrobbled for a bit, you might want to check the <a class="textlink" href="/admin_issues">Issues page</a> to see if you need to set up some rules. You can also manually add rules in your server's "rules" directory - just add your own .tsv file and read the instructions on how to declare a rule.
+	You can add some rules in your server's "rules" directory - just add your own .tsv file and read the instructions on how to declare a rule.
 	<br/><br/>
 
 	You can also set up some predefined rulesets right away!

maloja/web/jinja/artist.jinja

@@ -62,7 +62,7 @@
 			{% endif %}
 		</td>
 		<td class="text">
-			<h1 id="main_entity_name" class="headerwithextra">{{ info.artist }}</h1>
+			<h1 id="main_entity_name" class="headerwithextra">{{ info.artist | e }}</h1>
 				{% if competes %}<span class="rank"><a href="/charts_artists?max=100">#{{ info.position }}</a></span>{% endif %}
 				<br/>
 				{% if competes and included %}

maloja/web/jinja/partials/charts_artists_tiles.jinja

@@ -27,7 +27,7 @@
 						{% set rank = entry.rank %}
 						<td>
 							<a href="{{ links.url(artist) }}">
-							<div style='background-image:url("{{ images.get_artist_image(artist) }}")'>
+							<div class="lazy" data-bg="{{ images.get_artist_image(artist) }}"'>
 								<span class='stats'>#{{ rank }}</span> <span>{{ artist }}</span>
 							</div>
 							</a>

maloja/web/jinja/partials/charts_tracks_tiles.jinja

@@ -26,7 +26,7 @@
 						{% set rank = entry.rank %}
 						<td>
 							<a href="{{ links.url(track) }}">
-							<div style='background-image:url("{{ images.get_track_image(track) }}")'>
+							<div class="lazy" data-bg="{{ images.get_track_image(track) }}")'>
 								<span class='stats'>#{{ rank }}</span> <span>{{ track.title }}</span>
 							</div>
 							</a>

maloja/web/jinja/snippets/entityrow.jinja

@@ -8,7 +8,11 @@
 	{% set img = images.get_artist_image(entity) %}
 {% endif %}
 
-<td class='icon'><div style="background-image:url('{{ img }}')"></div></td>
+<td class='icon'>
+{% if settings['DISPLAY_ART_ICONS'] %}
+    <div class="lazy" data-bg="{{ img }}"></div>
+{% endif %}
+</td>
 {% if entity is mapping and 'artists' in entity %}
 	{% if settings['TRACK_SEARCH_PROVIDER'] %}
 		<td class='searchProvider'>{{ links.link_search(entity) }}</td>

maloja/web/jinja/snippets/links.jinja

@@ -5,7 +5,7 @@
 		{% set name = entity %}
 	{% endif %}
 
-	<a href="{{ url(entity) }}">{{ name }}</a>
+	<a href="{{ url(entity) }}">{{ name | e }}</a>
 {%- endmacro %}
 
 {% macro links(entities) -%}

maloja/web/jinja/snippets/pagination.jinja

@@ -10,8 +10,7 @@
 	{% if pages > 1 %}
 		{% if page > 1 %}
 			<a href='{{ mlj_uri.create_uri("",filterkeys,limitkeys,delimitkeys,amountkeys,{'page':0}) }}'>
-				<span class='stat_selector'>1</span>
-			</a> |
+				<span class='stat_selector'>1</span></a> |
 		{% endif %}
 
 		{% if page > 2 %}
@@ -20,8 +19,7 @@
 
 		{% if page > 0 %}
 			<a href='{{ mlj_uri.create_uri("",filterkeys,limitkeys,delimitkeys,amountkeys,{'page':page-1}) }}'>
-				<span class='stat_selector'>{{ page }}</span>
-			</a> «
+				<span class='stat_selector'>{{ page }}</span></a> «
 		{% endif %}
 
 		<span style='opacity:0.5;' class='stat_selector'>
@@ -30,8 +28,7 @@
 
 		{% if page < pages-1 %}
 			 » <a href='{{ mlj_uri.create_uri("",filterkeys,limitkeys,delimitkeys,amountkeys,{'page':page+1}) }}'>
-				 <span class='stat_selector'>{{ page+2 }}</span>
-			 </a>
+				 <span class='stat_selector'>{{ page+2 }}</span></a>
 		{% endif %}
 
 		{% if page < pages-3 %}
@@ -40,8 +37,7 @@
 
 		{% if page < pages-2 %}
 			 | <a href='{{ mlj_uri.create_uri("",filterkeys,limitkeys,delimitkeys,amountkeys,{'page':pages-1}) }}'>
-				 <span class='stat_selector'>{{ pages }}</span>
-			 </a>
+				 <span class='stat_selector'>{{ pages }}</span></a>
 		{% endif %}
 	{% endif %}
 

maloja/web/jinja/track.jinja

@@ -60,7 +60,7 @@
 		</td>
 		<td class="text">
 			<span>{{ links.links(track.artists) }}</span><br/>
-			<h1 id="main_entity_name" class="headerwithextra">{{ info.track.title }}</h1>
+			<h1 id="main_entity_name" class="headerwithextra">{{ info.track.title | e }}</h1>
 				{{ awards.certs(track) }}
 				<span class="rank"><a href="/charts_tracks?max=100">#{{ info.position }}</a></span>
 				<br/>

maloja/web/static/css/maloja.css

@@ -716,7 +716,7 @@ table.list td.amount {
 	text-align:right;
 }
 table.list td.bar {
-	width:500px;
+	width:400px;
 	/* background-color: var(--base-color); */
 	/* Remove 5er separators for bars */
 	/*border-color:rgba(0,0,0,0)!important;*/
@@ -734,7 +734,7 @@ table.list tr:hover td.bar div {
 }
 
 table.list td.chart {
-	width:500px;
+	width:400px;
 	/* background-color: var(--base-color); */
 	/* Remove 5er separators for bars */
 	/*border-color:rgba(0,0,0,0)!important;*/
@@ -848,8 +848,11 @@ table.tiles_top td div {
 
 table.tiles_top td span {
 	background-color:rgba(0,0,0,0.7);
-	display: table-cell;
+	display: inline-block;
+	margin-top:2%;
 	padding: 3px;
+	max-width: 67%;
+	vertical-align: text-top;
 }
 table.tiles_top td a:hover {
 	text-decoration: none;
@@ -863,12 +866,12 @@ table.tiles_1x1 td {
 table.tiles_2x2 td {
 	height:50%;
 	width:50%;
-	font-size:90%
+	font-size:80%
 }
 table.tiles_3x3 td {
 	height:33.333%;
 	width:33.333%;
-	font-size:70%
+	font-size:60%
 }
 table.tiles_4x4 td {
 	font-size:50%
@@ -877,6 +880,24 @@ table.tiles_5x5 td {
 	font-size:40%
 }
 
+/* Safari fix */
+table.tiles_sub.tiles_3x3 td div {
+        min-height: 100px;
+        min-width: 100px;
+}
+table.tiles_sub.tiles_2x2 td div {
+        min-height: 150px;
+        min-width: 150px;
+}
+table.tiles_sub.tiles_1x1 td div {
+        min-height: 300px;
+        min-width: 300px;
+}
+table.tiles_sub a span {
+        overflow-wrap: anywhere;
+}
+
+
 
 .summary_rank {
 	background-size:cover;

maloja/web/static/js/edit.js

@@ -104,7 +104,13 @@ function createTrackCell(trackinfo) {
 function editEntity() {
 
 	var namefield = document.getElementById('main_entity_name');
-	namefield.contentEditable = "plaintext-only";
+	try {
+		namefield.contentEditable = "plaintext-only"; // not supported by Firefox
+	}
+	catch (e) {
+		namefield.contentEditable = true;
+	}
+
 
 	namefield.addEventListener('keydown',function(e){
 		// dont allow new lines, done on enter

maloja/web/static/js/manualscrobble.js

@@ -126,14 +126,14 @@ function searchresult_manualscrobbling() {
 		console.log(tracks);
 		for (let t of tracks) {
 			track = document.createElement("span");
-			trackstr = t["artists"].join(", ") + " - " + t["title"];
+			trackstr = t.track["artists"].join(", ") + " - " + t.track["title"];
 			tracklink = t["link"];
 			track.innerHTML = "<a href='" + tracklink + "'>" +  trackstr + "</a>";
 			row = document.createElement("tr")
 			col1 = document.createElement("td")
 			button = document.createElement("button")
 			button.innerHTML = "Scrobble!"
-			button.onclick = function(){ scrobble(t["artists"],t["title"])};
+			button.onclick = function(){ scrobble(t.track["artists"],t.track["title"])};
 			col2 = document.createElement("td")
 			row.appendChild(col1)
 			col1.appendChild(button)

maloja/web/static/js/search.js

@@ -68,7 +68,7 @@ function searchresult() {
 			var node = oneresult.cloneNode(true);
 			node.setAttribute("onclick","goto('" + link + "')");
 			node.children[0].style.backgroundImage = "url('" + image + "')";
-			node.children[1].children[0].innerHTML = name;
+			node.children[1].children[0].textContent = name;
 
 			results_artists.appendChild(node);
 		}
@@ -82,8 +82,8 @@ function searchresult() {
 			var node = oneresult.cloneNode(true);
 			node.setAttribute("onclick","goto('" + link + "')");
 			node.children[0].style.backgroundImage = "url('" + image + "')";
-			node.children[1].children[0].innerHTML = artists;
-			node.children[1].children[2].innerHTML = title;
+			node.children[1].children[0].textContent = artists;
+			node.children[1].children[2].textContent = title;
 
 			results_tracks.appendChild(node);
 		}

maloja/web/static/txt/robots.txt

@@ -1,2 +1,2 @@
 User-agent: *
-Disallow: *
+Disallow: /

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "malojaserver"
-version = "3.1.1"
+version = "3.1.5"
 description = "Self-hosted music scrobble database"
 readme = "./README.md"
 requires-python = ">=3.7"
@@ -21,7 +21,7 @@ classifiers = [
 dependencies = [
 	"bottle>=0.12.16",
 	"waitress>=2.1.0",
-	"doreah>=1.9.2, <2",
+	"doreah>=1.9.4, <2",
 	"nimrodel>=0.8.0",
 	"setproctitle>=1.1.10",
 	#"pyvips>=2.1.16",

requirements.txt

@@ -1,6 +1,6 @@
 bottle>=0.12.16
 waitress>=2.1.0
-doreah>=1.9.2, <2
+doreah>=1.9.4, <2
 nimrodel>=0.8.0
 setproctitle>=1.1.10
 jinja2>=3.0.0
@@ -9,4 +9,3 @@ psutil>=5.8.0
 sqlalchemy>=1.4
 python-datauri>=1.1.0
 requests>=2.27.1
-

settings.md

@@ -1,4 +1,14 @@
-Technically, each setting can be set via environment variable or the settings file - simply add the prefix `MALOJA_` for environment variables. The columns are filled according to what is reasonable, it is recommended to use the settings file where possible and not configure each aspect of your server via environment variables!
+If you wish to adjust settings in the settings.ini file, do so while the server
+	is not running in order to avoid data being overwritten.
+
+	Technically, each setting can be set via environment variable or the settings
+	file - simply add the prefix `MALOJA_` for environment variables. It is recommended
+	to use the settings file where possible and not configure each aspect of your
+	server via environment variables!
+
+	You also can specify additional settings in the files`/run/secrets/maloja.yml` or
+	`/run/secrets/maloja.ini`, as well as their values directly in files of the respective
+	name in `/run/secrets/` (e.g. `/run/secrets/lastfm_api_key`).
 
 Settings File			| Environment Variable			| Type			| Description
 ------					| ---------						| ---------		| ---------
@@ -15,16 +25,14 @@ Settings File			| Environment Variable			| Type			| Description
 `logging`	| `MALOJA_LOGGING`  | Boolean | Enable Logging
 `dev_mode`	| `MALOJA_DEV_MODE`  | Boolean | Enable developer mode
 **Network**
-`host`	| `MALOJA_HOST`  | String | Host for your server - most likely :: for IPv6 or 0.0.0.0 for IPv4
+`host`	| `MALOJA_HOST`  | String | Host for your server, e.g. '*' for dual stack, '::' for IPv6 or '0.0.0.0' for IPv4
 `port`	| `MALOJA_PORT`  | Integer | Port
 **Technical**
 `cache_expire_positive`	| `MALOJA_CACHE_EXPIRE_POSITIVE`  | Integer | Days until images are refetched
 `cache_expire_negative`	| `MALOJA_CACHE_EXPIRE_NEGATIVE`  | Integer | Days until failed image fetches are reattempted
-`use_db_cache`	| `MALOJA_USE_DB_CACHE`  | Boolean | Use DB Cache
-`cache_database_short`	| `MALOJA_CACHE_DATABASE_SHORT`  | Boolean | Use volatile Database Cache
-`cache_database_perm`	| `MALOJA_CACHE_DATABASE_PERM`  | Boolean | Use permanent Database Cache
-`db_cache_entries`	| `MALOJA_DB_CACHE_ENTRIES`  | Integer | Maximal Cache entries
-`db_max_memory`	| `MALOJA_DB_MAX_MEMORY`  | Integer | Maximal percentage of RAM that should be used by whole system before Maloja discards cache entries. Use a higher number if your Maloja runs on a dedicated instance (e.g. a container)
+`db_max_memory`	| `MALOJA_DB_MAX_MEMORY`  | Integer | RAM Usage in percent at which Maloja should no longer increase its database cache.
+`use_request_cache`	| `MALOJA_USE_REQUEST_CACHE`  | Boolean | Use request-local DB Cache
+`use_global_cache`	| `MALOJA_USE_GLOBAL_CACHE`  | Boolean | Use global DB Cache
 **Fluff**
 `scrobbles_gold`	| `MALOJA_SCROBBLES_GOLD`  | Integer | How many scrobbles a track needs to be considered 'Gold' status
 `scrobbles_platinum`	| `MALOJA_SCROBBLES_PLATINUM`  | Integer | How many scrobbles a track needs to be considered 'Platinum' status
@@ -35,24 +43,33 @@ Settings File			| Environment Variable			| Type			| Description
 `scrobble_lastfm`	| `MALOJA_SCROBBLE_LASTFM`  | Boolean | Proxy-Scrobble to Last.fm
 `lastfm_api_key`	| `MALOJA_LASTFM_API_KEY`  | String | Last.fm API Key
 `lastfm_api_secret`	| `MALOJA_LASTFM_API_SECRET`  | String | Last.fm API Secret
+`lastfm_api_sk`	| `MALOJA_LASTFM_API_SK`  | String | Last.fm API Session Key
+`lastfm_username`	| `MALOJA_LASTFM_USERNAME`  | String | Last.fm Username
+`lastfm_password`	| `MALOJA_LASTFM_PASSWORD`  | String | Last.fm Password
 `spotify_api_id`	| `MALOJA_SPOTIFY_API_ID`  | String | Spotify API ID
 `spotify_api_secret`	| `MALOJA_SPOTIFY_API_SECRET`  | String | Spotify API Secret
 `audiodb_api_key`	| `MALOJA_AUDIODB_API_KEY`  | String | TheAudioDB API Key
+`other_maloja_url`	| `MALOJA_OTHER_MALOJA_URL`  | String | Other Maloja Instance URL
+`other_maloja_api_key`	| `MALOJA_OTHER_MALOJA_API_KEY`  | String | Other Maloja Instance API Key
 `track_search_provider`	| `MALOJA_TRACK_SEARCH_PROVIDER`  | String | Track Search Provider
 `send_stats`	| `MALOJA_SEND_STATS`  | Boolean | Send Statistics
+`proxy_images`	| `MALOJA_PROXY_IMAGES`  | Boolean | Whether third party images should be downloaded and served directly by Maloja (instead of just linking their URL)
 **Database**
 `invalid_artists`	| `MALOJA_INVALID_ARTISTS`  | Set | Artists that should be discarded immediately
 `remove_from_title`	| `MALOJA_REMOVE_FROM_TITLE`  | Set | Phrases that should be removed from song titles
 `delimiters_feat`	| `MALOJA_DELIMITERS_FEAT`  | Set | Delimiters used for extra artists, even when in the title field
 `delimiters_informal`	| `MALOJA_DELIMITERS_INFORMAL`  | Set | Delimiters in informal artist strings with spaces expected around them
 `delimiters_formal`	| `MALOJA_DELIMITERS_FORMAL`  | Set | Delimiters used to tag multiple artists when only one tag field is available
+`filters_remix`	| `MALOJA_FILTERS_REMIX`  | Set | Filters used to recognize the remix artists in the title
+`parse_remix_artists`	| `MALOJA_PARSE_REMIX_ARTISTS`  | Boolean | Parse Remix Artists
 **Web Interface**
 `default_range_charts_artists`	| `MALOJA_DEFAULT_RANGE_CHARTS_ARTISTS`  | Choice | Default Range Artist Charts
 `default_range_charts_tracks`	| `MALOJA_DEFAULT_RANGE_CHARTS_TRACKS`  | Choice | Default Range Track Charts
 `default_step_pulse`	| `MALOJA_DEFAULT_STEP_PULSE`  | Choice | Default Pulse Step
 `charts_display_tiles`	| `MALOJA_CHARTS_DISPLAY_TILES`  | Boolean | Display Chart Tiles
+`display_art_icons`	| `MALOJA_DISPLAY_ART_ICONS`  | Boolean | Display Album/Artist Icons
 `discourage_cpu_heavy_stats`	| `MALOJA_DISCOURAGE_CPU_HEAVY_STATS`  | Boolean | Prevent visitors from mindlessly clicking on CPU-heavy options. Does not actually disable them for malicious actors!
 `use_local_images`	| `MALOJA_USE_LOCAL_IMAGES`  | Boolean | Use Local Images
-`local_image_rotate`	| `MALOJA_LOCAL_IMAGE_ROTATE`  | Integer | Local Image Rotate
 `timezone`	| `MALOJA_TIMEZONE`  | Integer | UTC Offset
 `time_format`	| `MALOJA_TIME_FORMAT`  | String | Time Format
+`theme`	| `MALOJA_THEME`  | String | Theme