Copyright © 2012 Kaltura Inc.
All Rights Reserved. Designated trademarks and brands are the property of their respective owners.
Use of this web site constitutes acceptance of the Terms of Use and Privacy Policy.
EduVideo.org
Analytics Kaltura CE 5.0
Wed, 12/19/2012 - 12:10
Hello everybody!
I have an issue with the built in analytics tool. The only reports I get is in the content contribution part. Everything else is 0.
I really don't know what to do so if there is anyone with a fix/solution to this problem I would be more than gratefull.
Fri, 12/21/2012 - 14:29
#2
We have traced it this far atm the collect_res.sql is created but it's totally empty.
Fri, 01/11/2013 - 09:48
#3
In Kaltura CE 5.0, I'm not sure whether www_logs_insert.sh is used to collect statistics from apache logs. It uses two scripts /opt/kaltura/dwh/etlsource/execute/etl_hourly.sh and /opt/kaltura/dwh/etlsource/execute/etl_daily.sh to collect the data from kaltura_apache_access.log
I get both these scripts executed correctly and not able to find any errors. Still not getting data to the analytics.
Sat, 01/12/2013 - 04:43
#4
Try following ronileco's post at: http://ronileco.com/kaltura-ce5-analytics-13/
Or install v6 instead: https://github.com/kaltura/ce-packager/
Mon, 01/14/2013 - 07:19
#5
I've followed the http://ronileco.com/kaltura-ce5-analytics-13/ But I'm not getting any errors in the daily or hourly logs. It just got not getting any data.
I'm using ubuntu. Is it a problem due to apache access log format? Here are few lines from the kaltura apache access log. (Attached.)
kalturaCE5.abc.com:80 100.104.106.186 - - [14/Jan/2013:07:04:50 +0000] "GET //api_v3/index.php?service=stats&action=collect&kalsig=61f84db69a9db92fc39214cdef1e4bf2&event%3Aseek=false&event%3AclientVer=3%2E0%3Av3%2E5%2E35&event%3AsessionId=343865B3%2DBB9C%2D5C52%2D8132%2D37E002ED9C25&event%3ApartnerId=108&event%3AuiconfId=6709422&event%3AentryId=0%5Fgq06lgie&event%3AeventType=1&event%3Aduration=0&clientTag=kdp%3Av3%2E5%2E35%2Ccache%5Fst%3A1357817246&event%3AisFirstInSession=false&event%3AcurrentPoint=0&event%3AeventTimestamp=1358147115732&ignoreNull=1&event%3Areferrer=http%253A%2F%2Flocalhost%2Foffline%2Fkaltura%2Ehtml&ks=ZmQ1MzUxNzQ1MTQzOTAwYWM2ZjMzYThkODU2ZGFiZmVkMjg2NWE1MHwxMDg7MTA4OzEzNTgyMzM0ODU7MDsxMzU4MTQ3MDg1LjE3OTI7MDt2aWV3Oio7Ow%3D%3D&event%3AobjectType=KalturaStatsEvent HTTP/1.1" 200 471 "http://kalturace5.abc.com/index.php/kwidget/cache_st/1357817246/wid/_108/uiconf_id/6709422/entry_id/0_gq06lgie" "Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0" kalturaCE5.abc.com:80 100.104.106.186 - - [14/Jan/2013:07:04:50 +0000] "GET //api_v3/index.php?service=stats&action=collect&kalsig=dd2f60f944eff74d0235116b43bb9a61&event%3Aseek=false&event%3AclientVer=3%2E0%3Av3%2E5%2E35&event%3AsessionId=343865B3%2DBB9C%2D5C52%2D8132%2D37E002ED9C25&event%3ApartnerId=108&event%3AuiconfId=6709422&event%3AentryId=0%5Fgq06lgie&event%3AeventType=2&event%3Aduration=0&clientTag=kdp%3Av3%2E5%2E35%2Ccache%5Fst%3A1357817246&event%3AisFirstInSession=false&event%3AcurrentPoint=0&event%3AeventTimestamp=1358147115736&ignoreNull=1&event%3Areferrer=http%253A%2F%2Flocalhost%2Foffline%2Fkaltura%2Ehtml&ks=ZmQ1MzUxNzQ1MTQzOTAwYWM2ZjMzYThkODU2ZGFiZmVkMjg2NWE1MHwxMDg7MTA4OzEzNTgyMzM0ODU7MDsxMzU4MTQ3MDg1LjE3OTI7MDt2aWV3Oio7Ow%3D%3D&event%3AobjectType=KalturaStatsEvent HTTP/1.1" 200 473 "http://kalturace5.abc.com/index.php/kwidget/cache_st/1357817246/wid/_108/uiconf_id/6709422/entry_id/0_gq06lgie" "Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0" kalturaCE5.abc.com:80 100.104.106.186 - - [14/Jan/2013:07:04:50 +0000] "GET /p/108/sp/10800/playManifest/entryId/0_j1ep8khi/flavorId/0_0horq1a9/format/http/protocol/http/cdnHost/kalturaCE5.abc.com/ks/MjA2YTNiNmMyZDdhMThhZGRmYjBmN2JmZDViMjk3ODg5Y2E0OGRjNnwxMDg7MTA4OzEzNTgyMzM0ODU7MDsxMzU4MTQ3MDg1LjEyNjU7MDt2aWV3Oio7Ow==/uiConfId/6709422/a/a.f4m?referrer=aHR0cDovL2xvY2FsaG9zdC9vZmZsaW5lL2thbHR1cmEuaHRtbA== HTTP/1.1" 200 818 "http://kalturace5.abc.com/index.php/kwidget/cache_st/1357816952/wid/_108/uiconf_id/6709422/entry_id/0_j1ep8khi" "Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0" kalturaCE5.abc.com:80 127.0.0.1 - - [14/Jan/2013:07:04:52 +0000] "POST /api_v3/index.php?service=batch&action=getExclusiveAlmostDoneConvertJobs HTTP/1.1" 200 276 "-" "-" kalturaCE5.abc.com:80 127.0.0.1 - - [14/Jan/2013:07:04:53 +0000] "POST /api_v3/index.php?service=batch&action=getQueueSize HTTP/1.1" 200 274 "-" "-" kalturaCE5.abc.com:80 100.104.106.186 - - [14/Jan/2013:07:04:53 +0000] "GET //api_v3/index.php?service=stats&action=collect&kalsig=ba1a56994b87c570d4704447400f5e17&event%3AisFirstInSession=false&event%3AeventTimestamp=1358147118835&event%3AobjectType=KalturaStatsEvent&event%3AuiconfId=6709422&event%3AentryId=0%5Fj1ep8khi&ks=MjA2YTNiNmMyZDdhMThhZGRmYjBmN2JmZDViMjk3ODg5Y2E0OGRjNnwxMDg7MTA4OzEzNTgyMzM0ODU7MDsxMzU4MTQ3MDg1LjEyNjU7MDt2aWV3Oio7Ow%3D%3D&event%3Aseek=false&clientTag=kdp%3Av3%2E5%2E35%2Ccache%5Fst%3A1357816952&event%3AeventType=3&event%3Aduration=93&ignoreNull=1&event%3AcurrentPoint=40&event%3Areferrer=http%253A%2F%2Flocalhost%2Foffline%2Fkaltura%2Ehtml&event%3ApartnerId=108&event%3AsessionId=AAF96921%2D24B9%2DB3EA%2D886F%2D37E0001C0EDD&event%3AclientVer=3%2E0%3Av3%2E5%2E35 HTTP/1.1" 200 473 "http://kalturace5.abc.com/index.php/kwidget/cache_st/1357816952/wid/_108/uiconf_id/6709422/entry_id/0_j1ep8khi" "Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0" kalturaCE5.abc.com:80 127.0.0.1 - - [14/Jan/2013:07:04:54 +0000] "POST /api_v3/index.php?service=batch&action=getExclusiveAlmostDoneConvertProfileJobs HTTP/1.1" 200 276 "-" "-"
In the /opt/kaltura/dwh/cycles/process/ directory it is creating a copy of the split_kaltura_apache_access.log-20130111.00 but the files generated ds_bandwidth_usage_split_kaltura_apache_access.log-20130111.00 and ds_events_split_kaltura_apache_access.log-20130111.00 are empty.
Is there any other logs I need to check?
| Attachment | Size |
|---|---|
| kaltura_apache_access.log_.txt | 3.98 KB |
Mon, 01/14/2013 - 13:01
#6
In ubuntu, the default apache2.conf file set the LogFormat for vhost_combined as
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
which was overriding the LogFormat setting added in /opt/kaltura/app/configurations/apache/my_kaltura.conf
I've now commented out the apache2.conf LogFormat string. Now the kaltura_apache_access.log is populating properly.
Still I'm not getting data in analytics.
Tue, 01/15/2013 - 06:06
#7
Now I'm getting the logs as in the attached file. But still the statistics are not displayed. In the daily log now I get error
ERROR 14-01 16:37:09,071 - [[process file].[Insert events invalid ds line].insert_invalid_ds_line]. Insert / Update invalid lines.0 - Unable to commit connection(s) :org.pentaho.di.core.exception.KettleDatabaseException: Error comitting connection Communications link failure during commit(). Transaction resolution unknown.
cycle_id = 1
new_status = 'FAILED'
What can be the reason for this error?
| Attachment | Size |
|---|---|
| kaltura_apache_access.log_.txt | 4.02 KB |
Tue, 01/15/2013 - 09:55
#8
Hi,
Please try to reset this cycle by running the following command:
UPDATE kalturadw_ds.cycles set STATUS = 'GENERATED' where cycle_id = 1;
Now run the etl_hourly.sh manually or wait for the next run and please attach the hourly log to this thread.
Thanks,
Orly.
Tue, 01/15/2013 - 10:36
#9
I've set the status to GENERATED and executed the etl_hourly.sh. Attaching the log.
| Attachment | Size |
|---|---|
| etl_hourly-20130115-10.log_.txt | 261.72 KB |
Wed, 01/16/2013 - 12:02
#10
Sorry, Please ignore the last log attached. The error on log occurred when I tried to add the row level debugging enabled in the etl_hourly.sh. I've removed that and now the lock is working properly. But I get error when processing
the apache log.
INFO 16-01 11:19:35,187 - iterate file.0 - Opening file: /opt/kaltura/dwh/cycles/process /1/split_kaltura_apache_access.log-20130114.00 ERROR 16-01 11:20:48,353 - lookup entry.0 - Unexpected error : ERROR 16-01 11:20:48,578 - lookup entry.0 - org.pentaho.di.core.exception.KettleDatabaseException: Couldn't get field info from [SELECT * FROM dwh_dim_entries LIMIT 0]
I've tried changing the cycle status to GENERATED and execute the hourly script. I get the same error.
Attaching the full log. Please help.
Thanks.
| Attachment | Size |
|---|---|
| etl_hourly-20130116-11.log_.txt | 109.37 KB |
Wed, 01/16/2013 - 12:18
#11
Just wanted to ask you to change the log level :)
The broken pipe errors are MySQL related, please try to increase MySQL's wait_timeout variable
http://forums.pentaho.com/showthread.php?84393-Broken-pipe-error-while-l...
Thanks :)
Thu, 01/17/2013 - 04:25
#12
Thanks for the suggestions. I've increased the mysql wait timeout. Now both hourly and daily scripts executed properly and I'm getting the statistics.
Thank you very much.
Wed, 01/16/2013 - 12:08
#13
Now when ever I execute the etl_hourly.sh I get the error:
Full log attached to the previous post.
ERROR 15-01 17:13:48,886 - org.pentaho.di.trans.steps.update.Update
- org.pentaho.di.core.exception.KettleValueException:
LOCKED_STATE Integer : There was a data type error: the data type of
java.lang.Boolean object [false] does not correspond to value meta [Integer]
at org.pentaho.di.core.row.ValueMeta.getString(ValueMeta.java:1344)
at org.pentaho.di.core.row.RowMeta.getString(RowMeta.java:182)
at org.pentaho.di.core.row.RowMeta.getString(RowMeta.java:581)
at org.pentaho.di.trans.steps.update.Update.lookupValues(Update.java:131)
at org.pentaho.di.trans.steps.update.Update.processRow(Update.java:288)
at org.pentaho.di.trans.step.BaseStep.runStepThread(BaseStep.java:2889)
at org.pentaho.di.trans.steps.update.Update.run(Update.java:511)
DEBUG 15-01 17:13:48,886 - kalturadw_ds - Statement canceled!
DEBUG 15-01 17:13:48,886 - kalturadw_ds - Statement canceled!
DEBUG 15-01 17:13:48,887 - org.pentaho.di.trans.steps.update.Update - Signaling
'output done' to 0 output rowsets.
DEBUG 15-01 17:13:48,887 - kalturadw_ds - Rollback on database connection [kalturadw_ds]
DEBUG 15-01 17:13:48,887 - Lock is already seized.0 - Signaling 'output done' to 0 output rowsets.
INFO 15-01 17:13:48,889 - Lock is already seized.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=0)
DEBUG 15-01 17:13:48,889 - Get Now.0 - Signaling 'output done' to 1 output rowsets.
INFO 15-01 17:13:48,890 - Get Now.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
ERROR 15-01 17:13:48,887 - seize_lock_by_name - Errors detected!
ERROR 15-01 17:13:48,890 - seize_lock_by_name - Errors detected
Please help in fixing this issue and get the analytics working. Thanks.
[EDIT] This error was due to /level parameter to the etl_hourly.sh Now it is corrected. Still I get errors. Posted those to http://www.kaltura.org/analytics-kaltura-ce-50#comment-10297
Thu, 01/17/2013 - 04:41
#14
The analytics for Kaltura CE 5.0 is working now. Thank you orly.lampert for helping me get this working.
These are the steps I followed to get analytics working in ubuntu 11.10.
1) Commented out vhost_combined LogFormat in /etc/apache2/apache2.conf
2) Increased mysql wait_timeout in /etc/mysql/my.cnf
3) Restarted apache and mysql.
You may also have to check the in kalturadw_ds and make sure the locks are 0 in the lock table and and set the status from FAILED to GENERATED in the cycles table .
With these two steps, I started getting the statistics for all the videos uploaded after the change.
Sun, 04/14/2013 - 22:51
#15
Hello,
to solve this problem, I alter the table kalturadw_ds.locks to change lock_state from tinyint to int
alter table locks modify lock_state int(5);
And all work fine. I'm figthing with another problems now .. ;-)
j
I've installed Kaltura CE 5.0 in a Ubuntu 11.10, everything works fine except for the analytics. The dwh cron is running properly and logs are generated. The events are logged in the apache access logs. I'm not able to figure out the issue. The analytics only shows content contributions, everything else is empty.
Can any one point me the code that handle the parsing of access log and store the data in the kalturadw database? Am I missed anything in the apache configuration? or is there any plugin/application missed?
I found this http://blog.kaltura.org/data-makes-the-world-go-round post, but it is not saying how the data is collected.
The dwh logs shows no error. It would be great if some one point me the right direction.