Media Foundation/V4L2 grabber ... (#1119)

* - New Media Foundation grabber - JsonAPI available grabber fix - commented json config removed * Added libjpeg-turbo to dependencies * Fix OSX build Removed Azure Pipelines from build scripts * Remove Platform from Dashboard * Correct Grabber Namings * Grabber UI improvements, generic JSONEditor Selection Update * Active grabber fix * Stop Framebuffer grabber on failure * - Image format NV12 and I420 added - Flip mode - Scaling factor for MJPEG - VSCode (compile before run) - CI (push) dependency libjpeg-turbo added * Refactor MediaFoundation (Part 1) * Remove QDebug output * Added image flipping ability to MF Grabber * fix issue 1160 * -Reload MF Grabber only once per WebUI update - Cleanup * Improvements * - Set 'Software Frame Decimation' begin to 0 - Removed grabber specific device name from Log - Keep pixel format when switching resolution - Display 'Flip mode' correct in Log - BGR24 images always flipped * Refactor MediaFoundation (Part 2) * Refactor V4L2 grabber (part 1) (#62) * Media Foundation grabber adapted to V4L2 change * Enable Media Foundation grabber on windows * Have fps as int, fix height typo * Added video standards to JsonAPI output * Error handling in source reader improved * Fix "Frame to small" error * Discovery VideoSources and Dynamically Update Editor * Hide all element when no video grabber discovered, upate naming * Do not show unsupported grabbers * Copy Log to Clipboard * Update Grabber schema and Defaults * Update access levels and validate crop ranges * Height and width in Qt grabber corrected * Correct formatting * Untabify * Global component states across instances * Components divided on the dashboard * refactor * Fix Merge-issues * Database migration aligning with updated grabber model * Align Grabber.js with new utility functions * Allow editor-validation for enum-lists * Handle "Show Explainations scenario" correctly * Grabber - Ensure save is only possible on valid content * Dashboard update + fix GlobalSignal connection * Ensure default database is populated with current release * Correct grabber4L2 access level * Display Signal detection area in preview * Write Hyperion version into default config on compiling. * Create defaultconfig.json dynamically * WebUI changes * Correct grabber config look-ups * Refactor i18n language loading * Fix en.json * Split global capture from instance capture config * Update grabber default values * Standalone grabber: Add --debug switch * Enhance showInputOptionsForKey for multiple keys * Add grabber instance link to system grabber config * Only show signal detection area, if grabber is enabled * Always show Active element on grabber page * Remote control - Only display gabber status, if global grabber is enabled * WebUI optimization (thx to @mkcologne) Start Grabber only when global settings are enabled Fixed an issue in the WebUI preview * V4L2/MF changes * Jsoneditor, Correct translation for default values * Refactor LED-Device handling in UI and make element naming consistent * MF Discovery extended * Fix LGTM finding * Support Grabber Bri, Hue, Sat and Con in UI, plus their defaults * Concider Access level for item filtering * Concider Access level for item filtering * Revert "Concider Access level for item filtering" This reverts commit 5b0ce3c0f2. * Disable fpsSoftwareDecimation for framegrabber, as not supported yet * JSON-Editor- Add updated schema for validation on dynamic elements * added V4L2 color IDs * LGTM findings fix * destroy SR callback only on exit * Grabber.js - Hide elements not supported by platform * Fixed freezing start effect * Grabber UI - Hardware controls - Show current values and allow to reset to defaults * Grabber - Discovery - Add current values to properties * Small things * Clean-up Effects and have ENDLESS consistently defined * Fix on/off/on priority during startup, by initializing _prevVisComp in line with background priority * Add missing translation mappings * DirectX Grabber reactivated/ QT Grabber size decimation fixed * typo in push-master workflow * Use PreciseTimer for Grabber to ensure stable FPS timing * Set default Screencapture rate consistently * Fix libjpeg-turbo download * Remove Zero character from file * docker-compile Add PLATFORM parameter, only copy output file after successful compile * Framebuffer, Dispmanx, OSX, AML Grabber discovery, various clean-up and consistencies across grabbers * Fix merge problem - on docker-compile Add PLATFORM parameter, only copy output file after successful compile * Fix definition * OSXFRameGrabber - Revert cast * Clean-ups nach Feedback * Disable certain libraries when building armlogic via standard stretch image as developer * Add CEC availability to ServerInfo to have it platform independent * Grabber UI - Fix problem that crop values are not populated when refining editor rage * Preserve value when updating json-editor range * LEDVisualisation - Clear image when source changes * Fix - Preserve value when updating json-editor range * LEDVisualisation - Clear image when no component is active * Allow to have password handled by Password-Manager (#1263) * Update default signal detection area to green assuming rainbow grabber * LED Visualisation - Handle empty priority update * Fix yuv420 in v4l2 grabber * V4L2-Grabber discovery - Only report grabbers with valid video input information * Grabber - Update static variables to have them working in release build * LED Visualisation - ClearImage when no priorities * LED Visualisation - Fix Logo resizing issue * LED Visualisation - Have nearly black background and negative logo Co-authored-by: LordGrey <lordgrey.emmel@gmail.com> Co-authored-by: LordGrey <48840279+Lord-Grey@users.noreply.github.com>
2025-03-01 10:33:28 +00:00 · 2021-07-14 20:48:33 +02:00
parent b0e1510a78
commit c135d91986
163 changed files with 10756 additions and 5953 deletions
--- a/libsrc/grabber/qt/QtGrabber.cpp
+++ b/libsrc/grabber/qt/QtGrabber.cpp
@@ -7,23 +7,30 @@
 #include <QGuiApplication>
 #include <QWidget>
 #include <QScreen>
+#include <QJsonObject>
+#include <QJsonArray>
+#include <QJsonDocument>

-QtGrabber::QtGrabber(int cropLeft, int cropRight, int cropTop, int cropBottom, int pixelDecimation, int display)
-	: Grabber("QTGRABBER", 0, 0, cropLeft, cropRight, cropTop, cropBottom)
-	, _display(unsigned(display))
-	, _pixelDecimation(pixelDecimation)
-	, _screenWidth(0)
-	, _screenHeight(0)
-	, _src_x(0)
-	, _src_y(0)
-	, _src_x_max(0)
-	, _src_y_max(0)
-	, _screen(nullptr)
+// Constants
+namespace {
+const bool verbose = false;
+} //End of constants
+
+QtGrabber::QtGrabber(int display, int cropLeft, int cropRight, int cropTop, int cropBottom)
+	: Grabber("QTGRABBER", cropLeft, cropRight, cropTop, cropBottom)
+	  , _display(display)
+	  , _calculatedWidth(0)
+	  , _calculatedHeight(0)
+	  , _src_x(0)
+	  , _src_y(0)
+	  , _src_x_max(0)
+	  , _src_y_max(0)
+	  , _isWayland(false)
+	  , _screen(nullptr)
+	  , _isVirtual(false)
 {
+	_logger = Logger::getInstance("Qt");
 	_useImageResampler = false;
-
-	// init
-	setupDisplay();
 }

 QtGrabber::~QtGrabber()
@@ -36,51 +43,111 @@ void QtGrabber::freeResources()
 	// Qt seems to hold the ownership of the QScreen pointers
 }

+bool QtGrabber::open()
+{
+	bool rc = false;
+
+#ifndef _WIN32
+	if (getenv("WAYLAND_DISPLAY") != nullptr)
+	{
+		_isWayland = true;
+	}
+	else
+#endif
+	{
+		rc = true;
+	}
+	return rc;
+}
+
 bool QtGrabber::setupDisplay()
 {
-	// cleanup last screen
-	freeResources();
-
-	QScreen* primary = QGuiApplication::primaryScreen();
-	QList<QScreen *> screens = QGuiApplication::screens();
-	// inject main screen at 0, if not nullptr
-	if(primary != nullptr)
+	bool result = false;
+	if ( ! open() )
 	{
-		screens.prepend(primary);
-		// remove last main screen if twice in list
-		if(screens.lastIndexOf(primary) > 0)
-			screens.removeAt(screens.lastIndexOf(primary));
+		if ( _isWayland  )
+		{
+			Error(_log, "Grabber does not work under Wayland!");
+		}
 	}
-
-	if(screens.isEmpty())
+	else
 	{
-		Error(_log, "No displays found to capture from!");
-		return false;
+		// cleanup last screen
+		freeResources();
+		_numberOfSDisplays = 0;
+
+		QScreen* primary = QGuiApplication::primaryScreen();
+		QList<QScreen *> screens = QGuiApplication::screens();
+		// inject main screen at 0, if not nullptr
+		if(primary != nullptr)
+		{
+			screens.prepend(primary);
+			// remove last main screen if twice in list
+			if(screens.lastIndexOf(primary) > 0)
+			{
+				screens.removeAt(screens.lastIndexOf(primary));
+			}
+		}
+
+		if(screens.isEmpty())
+		{
+			Error(_log, "No displays found to capture from!");
+			result =  false;
+		}
+		else
+		{
+			_numberOfSDisplays = screens.size();
+
+			Info(_log,"Available Displays:");
+			int index = 0;
+			for(auto * screen : qAsConst(screens))
+			{
+				const QRect geo = screen->geometry();
+				Info(_log,"Display %d: Name: %s Geometry: (L,T,R,B) %d,%d,%d,%d Depth:%dbit", index, QSTRING_CSTR(screen->name()), geo.left(), geo.top() ,geo.right(), geo.bottom(), screen->depth());
+				++index;
+			}
+
+			if (screens.at(0)->size() != screens.at(0)->virtualSize())
+			{
+				const QRect vgeo = screens.at(0)->virtualGeometry();
+				Info(_log,"Display %d: Name: %s Geometry: (L,T,R,B) %d,%d,%d,%d Depth:%dbit", _numberOfSDisplays, "All Displays", vgeo.left(), vgeo.top() ,vgeo.right(), vgeo.bottom(), screens.at(0)->depth());
+			}
+
+			_isVirtual = false;
+			// be sure the index is available
+			if (_display > _numberOfSDisplays - 1 )
+			{
+
+				if ((screens.at(0)->size() != screens.at(0)->virtualSize()) && (_display == _numberOfSDisplays))
+				{
+					_isVirtual = true;
+					_display = 0;
+
+				}
+				else
+				{
+					Info(_log, "The requested display index '%d' is not available, falling back to display 0", _display);
+					_display = 0;
+				}
+			}
+
+			// init the requested display
+			_screen = screens.at(_display);
+			connect(_screen, &QScreen::geometryChanged, this, &QtGrabber::geometryChanged);
+			updateScreenDimensions(true);
+
+			if (_isVirtual)
+			{
+				Info(_log, "Using virtual display across all screens");
+			}
+			else
+			{
+				Info(_log,"Initialized display %d", _display);
+			}
+			result =  true;
+		}
 	}
-
-	Info(_log,"Available Displays:");
-	int index = 0;
-	for(auto screen : screens)
-	{
-		const QRect geo = screen->geometry();
-		Info(_log,"Display %d: Name:%s Geometry: (L,T,R,B) %d,%d,%d,%d Depth:%dbit", index, QSTRING_CSTR(screen->name()), geo.left(), geo.top() ,geo.right(), geo.bottom(), screen->depth());
-		index++;
-	}
-
-	// be sure the index is available
-	if(_display > unsigned(screens.size()-1))
-	{
-		Info(_log, "The requested display index '%d' is not available, falling back to display 0", _display);
-		_display = 0;
-	}
-
-	// init the requested display
-	_screen = screens.at(_display);
-	connect(_screen, &QScreen::geometryChanged, this, &QtGrabber::geometryChanged);
-	updateScreenDimensions(true);
-
-	Info(_log,"Initialized display %d", _display);
-	return true;
+	return result;
 }

 void QtGrabber::geometryChanged(const QRect &geo)
@@ -91,90 +158,109 @@ void QtGrabber::geometryChanged(const QRect &geo)

 int QtGrabber::grabFrame(Image<ColorRgb> & image)
 {
-	if (!_enabled) return 0;
-	if(_screen == nullptr)
+	int rc = 0;
+	if (_isEnabled && !_isDeviceInError)
 	{
-		// reinit, this will disable capture on failure
-		setEnabled(setupDisplay());
-		return -1;
-	}
-	QPixmap originalPixmap = _screen->grabWindow(0, _src_x, _src_y, _src_x_max, _src_y_max);
-	QPixmap resizedPixmap = originalPixmap.scaled(_width,_height);
-	QImage imageFrame = resizedPixmap.toImage().convertToFormat( QImage::Format_RGB888);
-	image.resize(imageFrame.width(), imageFrame.height());
-
-	for (int y=0; y<imageFrame.height(); ++y)
-		for (int x=0; x<imageFrame.width(); ++x)
+		if(_screen == nullptr)
 		{
-			QColor inPixel(imageFrame.pixel(x,y));
-			ColorRgb & outPixel = image(x,y);
-			outPixel.red   = inPixel.red();
-			outPixel.green = inPixel.green();
-			outPixel.blue  = inPixel.blue();
+			// reinit, this will disable capture on failure
+			bool result = setupDisplay();
+			setEnabled(result);
 		}

-	return 0;
+		if (_isEnabled)
+		{
+			QPixmap originalPixmap = _screen->grabWindow(0, _src_x, _src_y, _src_x_max, _src_y_max);
+			if (originalPixmap.isNull())
+			{
+				rc = -1;
+			}
+			else
+			{
+				QImage imageFrame = originalPixmap.toImage().scaled(_calculatedWidth, _calculatedHeight).convertToFormat( QImage::Format_RGB888);
+				image.resize(static_cast<uint>(_calculatedWidth), static_cast<uint>(_calculatedHeight));
+
+				for (int y = 0; y < imageFrame.height(); y++)
+				{
+					memcpy((unsigned char*)image.memptr() + y * image.width() * 3, static_cast<unsigned char*>(imageFrame.scanLine(y)), imageFrame.width() * 3);
+				}
+			}
+		}
+	}
+	return rc;
 }

 int QtGrabber::updateScreenDimensions(bool force)
 {
-	if(!_screen)
+	if(_screen == nullptr)
+	{
 		return -1;
+	}

-	const QRect& geo = _screen->geometry();
-	if (!force && _screenWidth == unsigned(geo.right()) && _screenHeight == unsigned(geo.bottom()))
+	QRect geo;
+
+	if (_isVirtual)
+	{
+		geo = _screen->virtualGeometry();
+	}
+	else
+	{
+		geo = _screen->geometry();
+	}
+	if (!force && _width == geo.width() && _height == geo.height())
 	{
 		// No update required
 		return 0;
 	}

-	Info(_log, "Update of screen resolution: [%dx%d] to [%dx%d]", _screenWidth, _screenHeight, geo.right(), geo.bottom());
-	_screenWidth  = geo.right() - geo.left();
-	_screenHeight = geo.bottom() - geo.top();
+	Info(_log, "Update of screen resolution: [%dx%d] to [%dx%d]", _width, _height, geo.width(), geo.height());
+	_width  = geo.width();
+	_height = geo.height();

-	int width=0, height=0;
+	int width=0;
+	int height=0;

 	// Image scaling is performed by Qt
-	width  =  (_screenWidth > unsigned(_cropLeft + _cropRight))
-		? ((_screenWidth - _cropLeft - _cropRight) / _pixelDecimation)
-		: (_screenWidth / _pixelDecimation);
+	width  =  (_width > (_cropLeft + _cropRight))
+				? ((_width - _cropLeft - _cropRight) / _pixelDecimation)
+				: (_width / _pixelDecimation);

-	height =  (_screenHeight > unsigned(_cropTop + _cropBottom))
-		? ((_screenHeight - _cropTop - _cropBottom) / _pixelDecimation)
-		: (_screenHeight / _pixelDecimation);
+	height =  (_height > (_cropTop + _cropBottom))
+				 ? ((_height - _cropTop - _cropBottom) / _pixelDecimation)
+				 : (_height / _pixelDecimation);


 	// calculate final image dimensions and adjust top/left cropping in 3D modes
 	switch (_videoMode)
 	{
 	case VideoMode::VIDEO_3DSBS:
-		_width  = width /2;
-		_height = height;
+		_calculatedWidth  = width /2;
+		_calculatedHeight = height;
 		_src_x  = _cropLeft / 2;
 		_src_y  = _cropTop;
-		_src_x_max = (_screenWidth / 2) - _cropRight;
-		_src_y_max = _screenHeight - _cropBottom;
+		_src_x_max = (_width / 2) - _cropRight - _cropLeft;
+		_src_y_max = _height - _cropBottom - _cropTop;
 		break;
 	case VideoMode::VIDEO_3DTAB:
-		_width  = width;
-		_height = height / 2;
+		_calculatedWidth  = width;
+		_calculatedHeight = height / 2;
 		_src_x  = _cropLeft;
 		_src_y  = _cropTop / 2;
-		_src_x_max = _screenWidth - _cropRight;
-		_src_y_max = (_screenHeight / 2) - _cropBottom;
+		_src_x_max = _width - _cropRight - _cropLeft;
+		_src_y_max = (_height / 2) - _cropBottom - _cropTop;
 		break;
 	case VideoMode::VIDEO_2D:
 	default:
-		_width  = width;
-		_height = height;
+		_calculatedWidth  = width;
+		_calculatedHeight = height;
 		_src_x  = _cropLeft;
 		_src_y  = _cropTop;
-		_src_x_max = _screenWidth - _cropRight;
-		_src_y_max = _screenHeight - _cropBottom;
+		_src_x_max = _width - _cropRight - _cropLeft;
+		_src_y_max = _height - _cropBottom - _cropTop;
 		break;
 	}

-	Info(_log, "Update output image resolution to [%dx%d]", _width, _height);
+	Info(_log, "Update output image resolution to [%dx%d]", _calculatedWidth, _calculatedHeight);
 	return 1;
 }

@@ -184,22 +270,129 @@ void QtGrabber::setVideoMode(VideoMode mode)
 	updateScreenDimensions(true);
 }

-void QtGrabber::setPixelDecimation(int pixelDecimation)
+bool QtGrabber::setPixelDecimation(int pixelDecimation)
 {
-	_pixelDecimation = pixelDecimation;
+	bool rc (true);
+	if(Grabber::setPixelDecimation(pixelDecimation))
+	{
+		if ( updateScreenDimensions(true) < 0)
+		{
+			rc = false;
+		}
+	}
+	return rc;
 }

-void QtGrabber::setCropping(unsigned cropLeft, unsigned cropRight, unsigned cropTop, unsigned cropBottom)
+void QtGrabber::setCropping(int cropLeft, int cropRight, int cropTop, int cropBottom)
 {
 	Grabber::setCropping(cropLeft, cropRight, cropTop, cropBottom);
 	updateScreenDimensions(true);
 }

-void QtGrabber::setDisplayIndex(int index)
+bool QtGrabber::setDisplayIndex(int index)
 {
-	if(_display != unsigned(index))
+	bool rc (true);
+	if (_display != index)
 	{
-		_display = unsigned(index);
-		setupDisplay();
+		if (index <= _numberOfSDisplays)
+		{
+			_display = index;
+		}
+		else {
+			_display = 0;
+		}
+		rc = setupDisplay();
 	}
+	return rc;
+}
+
+QJsonObject QtGrabber::discover(const QJsonObject& params)
+{
+	DebugIf(verbose, _log, "params: [%s]", QString(QJsonDocument(params).toJson(QJsonDocument::Compact)).toUtf8().constData());
+
+	QJsonObject inputsDiscovered;
+	if ( open() )
+	{
+		QList<QScreen*> screens = QGuiApplication::screens();
+		if (!screens.isEmpty())
+		{
+			inputsDiscovered["device"] = "qt";
+			inputsDiscovered["device_name"] = "QT";
+			inputsDiscovered["type"] = "screen";
+
+			QJsonArray video_inputs;
+			QJsonArray fps = { 1, 5, 10, 15, 20, 25, 30, 40, 50, 60 };
+
+			for (int i = 0; i < screens.size(); ++i)
+			{
+				QJsonObject in;
+
+				QString name = screens.at(i)->name();
+				int pos = name.lastIndexOf('\\');
+				if (pos != -1)
+				{
+					name = name.right(name.length()-pos-1);
+				}
+
+				in["name"] = name;
+				in["inputIdx"] = i;
+
+				QJsonArray formats;
+				QJsonObject format;
+
+				QJsonArray resolutionArray;
+
+				QJsonObject resolution;
+
+				resolution["width"] = screens.at(i)->size().width();
+				resolution["height"] = screens.at(i)->size().height();
+				resolution["fps"] = fps;
+
+				resolutionArray.append(resolution);
+
+				format["resolutions"] = resolutionArray;
+				formats.append(format);
+
+				in["formats"] = formats;
+				video_inputs.append(in);
+			}
+
+			if (screens.at(0)->size() != screens.at(0)->virtualSize())
+			{
+				QJsonObject in;
+				in["name"] = "All Displays";
+				in["inputIdx"] = screens.size();
+				in["virtual"] = true;
+
+				QJsonArray formats;
+				QJsonObject format;
+
+				QJsonArray resolutionArray;
+
+				QJsonObject resolution;
+
+				resolution["width"] = screens.at(0)->virtualSize().width();
+				resolution["height"] = screens.at(0)->virtualSize().height();
+				resolution["fps"] = fps;
+
+				resolutionArray.append(resolution);
+
+				format["resolutions"] = resolutionArray;
+				formats.append(format);
+
+				in["formats"] = formats;
+				video_inputs.append(in);
+			}
+			inputsDiscovered["video_inputs"] = video_inputs;
+		}
+
+		if (inputsDiscovered.isEmpty())
+		{
+			DebugIf(verbose, _log, "No displays found to capture from!");
+		}
+	}
+	DebugIf(verbose, _log, "device: [%s]", QString(QJsonDocument(inputsDiscovered).toJson(QJsonDocument::Compact)).toUtf8().constData());
+
+	return inputsDiscovered;
+
 }
--- a/libsrc/grabber/qt/QtWrapper.cpp
+++ b/libsrc/grabber/qt/QtWrapper.cpp
@@ -1,9 +1,20 @@
 #include <grabber/QtWrapper.h>

-QtWrapper::QtWrapper(int cropLeft, int cropRight, int cropTop, int cropBottom, int pixelDecimation, int display, unsigned updateRate_Hz)
-	: GrabberWrapper("Qt", &_grabber, 0, 0, updateRate_Hz)
-	, _grabber(cropLeft, cropRight, cropTop, cropBottom, pixelDecimation, display)
-{}
+QtWrapper::QtWrapper( int updateRate_Hz,
+					  int display,
+					  int pixelDecimation,
+					  int cropLeft, int cropRight, int cropTop, int cropBottom
+					  )
+	: GrabberWrapper("Qt", &_grabber, updateRate_Hz)
+	  , _grabber(display, cropLeft, cropRight, cropTop, cropBottom)
+{
+	_grabber.setPixelDecimation(pixelDecimation);
+}
+
+bool QtWrapper::open()
+{
+	return _grabber.open();
+}

 void QtWrapper::action()
 {